We are in the process of merging smaller example code repositories into larger parent repositories on the MDN Web Docs project. While we thought that copying the files from one repository into the new one would lose commit history, we felt that this might be an OK strategy. After all, we are not deleting the old repository but archiving it.
After having moved a few of these, we did receive an issue from a community member stating that it is not ideal to lose history while moving these repositories and that there could be a relatively simple way to avoid this. I experimented with a couple of different options and finally settled on a strategy based on the one shared by Eric Lee on his blog.
tl;dr The approach is to use basic git commands to apply all of the histories of our old repo onto a new repo without needing special tooling.
This is how Eric describes the first steps:
# Assume the current directory is where we want the new repository to be created # Create the new repository git init # Before we do a merge, we need to have an initial commit, so we’ll make a dummy commit dir > deleteme.txt git add . git commit -m “Initial dummy commit” # Add a remote for and fetch the old repo git remote add -f old_a <OldA repo URL> # Merge the files from old_a/master into new/master git merge old_a/master
I could skip everything up to the
git remote ... step as my target repository already had some history, so I started as follows:
git clone https://github.com/mdn/dom-examples.git cd dom-examples
git log on this repository, I see the following commit history:
commit cdfd2aeb93cb4bd8456345881997fcec1057efbb (HEAD -> master, upstream/master) Merge: 1c7ff6e dfe991b Author: Date: Fri Aug 5 10:21:27 2022 +0200 Merge pull request #143 from mdn/sideshowbarker/webgl-sample6-UNPACK_FLIP_Y_WEBGL “Using textures in WebGL”: Fix orientation of Firefox logo commit dfe991b5d1b34a492ccd524131982e140cf1e555 Author: Date: Fri Aug 5 17:08:50 2022 +0900 “Using textures in WebGL”: Fix orientation of Firefox logo Fixes <https://github.com/mdn/content/issues/10132> commit 1c7ff6eec8bb0fff5630a66a32d1b9b6b9d5a6e5 Merge: be41273 5618100 Author: Date: Fri Aug 5 09:01:56 2022 +0200 Merge pull request #142 from mdn/sideshowbarker/webgl-demo-add-playsInline-drop-autoplay WebGL sample8: Drop “autoplay”; add “playsInline” commit 56181007b7a33907097d767dfe837bb5573dcd38 Author: Date: Fri Aug 5 13:41:45 2022 +0900
With the current setup, I could continue from the
git remote command, but I wondered if the current directory contained files or folders that would conflict with those in the service worker repository. I searched around some more to see if anyone else had run into this same situation but did not find an answer. Then it hit me! I need to prepare the service worker repo to be moved.
What do I mean by that? I need to create a new directory in the root of the
sw-test repo called
service-worker/sw-test and move all relevant files into this new subdirectory. This will allow me to safely merge it into
dom-examples as everything is contained in a subfolder already.
To get started, I need to clone the repo we want to merge into
git clone https://github.com/mdn/sw-test.git cd sw-test
Ok, now we can start preparing the repo. The first step is to create our new subdirectory.
mkdir service-worker mkdir service-worker/sw-test
NOTE: Do not yet run any of the commands below at this stage.
# enable extendedglob for ZSH set -o extendedglob mv ^sw-test(D) service-worker/swtest
The above command is a little more complex than you might think. It uses a negation syntax. The next section explains why we need it and how to enable it.
How to exclude subdirectories when using
While the end goal seemed simple, I am pretty sure I grew a small animal’s worth of grey hair trying to figure out how to make that last move command work. I read many StackOverflow threads, blog posts, and manual pages for the different commands with varying amounts of success. However, none of the initial set of options quite met my needs. I finally stumbled upon two StackOverflow threads that brought me to the answer.
To spare you the trouble, here is what I had to do.
First, a note. I am on a Mac using ZSH (since macOS Catalina, this is now the default shell). Depending on your shell, the instructions below may differ.
For new versions of ZSH, you use the
set -o and
set +o commands to enable and disable settings. To enable
extendedglob, I used the following command:
# Yes, this _enables_ it set -o extendedglob
On older versions of ZSH, you use the
bash, you can achieve the same using the following command:
shopt -s extglob
Why do you even have to do this, you may ask? Without this, you will not be able to use the negation operator I use in the above move command, which is the crux of the whole thing. If you do the following, for example:
mkdir service-worker mv * service-worker/sw-test
It will “work,” but you will see an error message like this:
mv: rename service-worker to service-worker/sw-test/service-worker: Invalid argument
We want to tell the operating system to move everything into our new subfolder except the subfolder itself. We, therefore, need this negation syntax. It is not enabled by default because it could cause problems if file names contain some of the
extendedglob patterns, such as
^. So we need to enable it explicitly.
NOTE: You might also want to disable it after completing your move operation.
Now that we know how and why we want
extendedglob enabled, we move on to using our new powers.
NOTE: Do not yet run any of the commands below at this stage.
mv ^sw-test(D) service-worker/sw-test
The above means:
- Move all the files in the current directory into
- Do not try to move the
- The (D) option tells the move command to also move all hidden files, such as
.gitignore, and hidden folders, such as
NOTE: I found that if I typed
mv ^sw-testand pressed tab, my terminal would expand the command to
mv CODE_OF_CONDUCT.md LICENSE README.md app.js gallery image-list.js index.html service-worker star-wars-logo.jpg style.css sw.js.If I typed
mv ^sw-test(D)and pressed tab, it would expand to
mv .git .prettierrc CODE_OF_CONDUCT.md LICENSE README.md app.js gallery image-list.js index.html service-worker star-wars-logo.jpg style.css sw.js. This is interesting because it clearly demonstrates what happens under the hood. This allows you to see the effect of using
(D)clearly. I am not sure whether this is just a native ZSH thing or one of my terminal plugins, such as Fig. Your mileage may vary.
Handling hidden files and creating a pull request
While it is nice to be able to move all of the hidden files and folders like this, it causes a problem. Because the
.git folder is transferred into our new subfolder, our root directory is no longer seen as a Git repository. This is a problem.
Therefore, I will not run the above command with
(D) but instead move the hidden files as a separate step. I will run the following command instead:
mv ^(sw-test|service-worker) service-worker/sw-test
At this stage, if you run
ls it will look like it moved everything. That is not the case because the
ls command does not list hidden files. To do that, you need to pass the
-A flag as shown below:
You should now see something like the following:
❯ ls -A .git .prettierrc service-worker
Looking at the above output, I realized that I should not need to move the
.git folder. All I needed to do now was to run the following command:
mv .prettierrc service-worker
After running the above command,
ls -A will now output the following:
❯ ls -A .git simple-service-worker
Time to do a little celebration dance 😁
We can move on now that we have successfully moved everything into our new subdirectory. However, while doing this, I realized I forgot to create a feature branch for the work.
Not a problem. I just run the command,
git switch -C prepare-repo-for-move. Running
git status at this point should output something like this:
❯ git status On branch prepare-repo-for-move Changes not staged for commit: (use "git add/rm <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) deleted: .prettierrc deleted: CODE_OF_CONDUCT.md deleted: LICENSE deleted: README.md deleted: app.js deleted: gallery/bountyHunters.jpg deleted: gallery/myLittleVader.jpg deleted: gallery/snowTroopers.jpg deleted: image-list.js deleted: index.html deleted: star-wars-logo.jpg deleted: style.css deleted: sw.js Untracked files: (use "git add <file>..." to include in what will be committed) service-worker/ no changes added to commit (use "git add" and/or "git commit -a")
Great! Let’s add our changes and commit them.
git add . git commit -m 'Moved all source files into new subdirectory'
Now we want to push our changes and open a pull request.
Woop! Let’s push:
git push origin prepare-repo-for-move
Head over to your repository on GitHub. You should see a banner like “mv-files-into-subdir had recent pushes less than a minute ago” and a “Compare & pull request” button.
Click the button and follow the steps to open the pull request. Once the pull request is green and ready to merge, go ahead and merge!
NOTE: Depending on your workflow, this is the point to ask a team member to review your proposed changes before merging. It is also a good idea to have a look over the changes in the “Files changed” tab to ensure nothing is part of the pull request you did not intend. If any conflicts prevent your pull request from being merged, GitHub will warn you about these, and you will need to resolve them. This can be done directly on GitHub.com or locally and pushed to GitHub as a separate commit.
When you head back to the code view on GitHub, you should see our new subdirectory and the
With that, our repository is ready to move.
Merging our repositories
Back in the terminal, you want to switch back to the
git switch main
You can now safely delete the feature branch and pull down the changes from your remote.
git branch -D prepare-repo-for-move git pull origin main
ls -A after pulling the latest should now show the following:
❯ ls -A .git README.md service-worker
git log in the root outputs the following:
commit 8fdfe7379130b8d6ea13ea8bf14a0bb45ad725d0 (HEAD -> gh-pages, origin/gh-pages, origin/HEAD) Author: Schalk Neethling Date: Thu Aug 11 22:56:48 2022 +0200 Create README.md commit 254a95749c4cc3d7d2c7ec8a5902bea225870176 Merge: f5c319b bc2cdd9 Author: Schalk Neethling Date: Thu Aug 11 22:55:26 2022 +0200 Merge pull request #45 from mdn/prepare-repo-for-move chore: prepare repo for move to dom-examples commit bc2cdd939f568380ce03d56f50f16f2dc98d750c (origin/prepare-repo-for-move) Author: Schalk Neethling Date: Thu Aug 11 22:53:13 2022 +0200 chore: prepare repo for move to dom-examples Prepping the repository for the move to dom-examples commit f5c319be3b8d4f14a1505173910877ca3bb429e5 Merge: d587747 2ed0eff Author: Ruth John Date: Fri Mar 18 12:24:09 2022 +0000 Merge pull request #43 from SimonSiefke/add-navigation-preload
Here are the commands left over from where we diverted earlier on.
# Add a remote for and fetch the old repo git remote add -f old_a <OldA repo URL> # Merge the files from old_a/master into new/master git merge old_a/master
Alrighty, let’s wrap this up. First, we need to move into the root of the project to which we want to move our project. For our purpose here, this is the
dom-examples directory. Once in the root of the directory, run the following:
git remote add -f swtest https://github.com/mdn/sw-test.git
-ftells Git to fetch the remote branches. The
sswis a name you give to the remote so this could really be anything.
After running the command, I got the following output:
❯ git remote add -f swtest https://github.com/mdn/sw-test.git Updating swtest remote: Enumerating objects: 500, done. remote: Counting objects: 100% (75/75), done. remote: Compressing objects: 100% (57/57), done. remote: Total 500 (delta 35), reused 45 (delta 15), pack-reused 425 Receiving objects: 100% (500/500), 759.76 KiB | 981.00 KiB/s, done. Resolving deltas: 100% (269/269), done. From <https://github.com/mdn/sw-test> * [new branch] gh-pages -> swtest/gh-pages * [new branch] master -> swtest/master * [new branch] move-prettierrc -> swtest/move-prettierrc * [new branch] rename-sw-test -> swtest/rename-sw-test
NOTE: While we deleted the branch locally, this is not automatically synced with the remote, so this is why you will still see a reference to the
rename-sw-testbranch. If you wanted to delete it on the remote, you would run the following from the root of that repository:
git push origin :rename-sw-test(if you have configured your repository “to automatically delete head branches”, this will be automatically deleted for you)
Only a few commands left.
NOTE: Do not yet run any of the commands below at this stage.
git merge swtest/gh-pages
Whoops! When I ran the above, I got the following error:
❯ git merge swtest/gh-pages fatal: refusing to merge unrelated histories
But this is pretty much exactly what I do want, right? This is the default behavior of the
merge command, but you can pass a flag and allow this behavior.
git merge swtest/gh-pages --allow-unrelated-histories
gh-pages? More often than not, the one you will merge here will be
mainbut for this particular repository, the default branch was named
gh-pages. It used to be that when using GitHub pages, you would need a branch called
gh-pagesthat will then be automatically deployed by GitHub to a URL that would be something like mdn.github.io/sw-test.
After running the above, I got the following:
❯ git merge swtest/gh-pages --allow-unrelated-histories Auto-merging README.md CONFLICT (add/add): Merge conflict in README.md Automatic merge failed; fix conflicts and then commit the result.
Ah yes, of course. Our current project and the one we are merging both contain a
README.md, so Git is asking us to decide what to do. If you open up the
README.md file in your editor, you will notice something like this:
<<<<<<< HEAD =======
There might be a number of these in the file. You will also see some entries like this,
>>>>>>> swtest/gh-pages. This highlights the conflicts that Git is not sure how to resolve. You could go through and clear these manually. In this instance, I just want what is in the
README.md at the root of the
dom-examples repo, so I will clean up the conflicts or copy the content from the
README.md from GitHub.
As Git requested, we will add and commit our changes.
git add . git commit -m 'merging sw-test into dom-examples'
The above resulted in the following output:
❯ git commit [146-chore-move-sw-test-into-dom-examples 4300221] Merge remote-tracking branch 'swtest/gh-pages' into 146-chore-move-sw-test-into-dom-examples
If I now run
git log in the root of the directory, I see the following:
commit 4300221fe76d324966826b528f4a901c5f17ae20 (HEAD -> 146-chore-move-sw-test-into-dom-examples) Merge: cdfd2ae 70c0e1e Author: Schalk Neethling Date: Sat Aug 13 14:02:48 2022 +0200 Merge remote-tracking branch 'swtest/gh-pages' into 146-chore-move-sw-test-into-dom-examples commit 70c0e1e53ddb7d7a26e746c4a3412ccef5a683d3 (swtest/gh-pages) Merge: 4b7cfb2 d4a042d Author: Schalk Neethling Date: Sat Aug 13 13:30:58 2022 +0200 Merge pull request #47 from mdn/move-prettierrc chore: move prettierrc commit d4a042df51ab65e60498e949ffb2092ac9bccffc (swtest/move-prettierrc) Author: Schalk Neethling Date: Sat Aug 13 13:29:56 2022 +0200 chore: move prettierrc Move `.prettierrc` into the siple-service-worker folder commit 4b7cfb239a148095b770602d8f6d00c9f8b8cc15 Merge: 8fdfe73 c86d1a1 Author: Schalk Neethling Date: Sat Aug 13 13:22:31 2022 +0200 Merge pull request #46 from mdn/rename-sw-test
Yahoooo! That is the history from
sw-test now in our current repository! Running
ls -A now shows me:
❯ ls -A .git indexeddb-examples screen-wake-lock-api .gitignore insert-adjacent screenleft-screentop CODE_OF_CONDUCT.md matchmedia scrolltooptions LICENSE media server-sent-events README.md media-session service-worker abort-api mediaquerylist streams auxclick payment-request touchevents canvas performance-apis web-animations-api channel-messaging-basic picture-in-picture web-crypto channel-messaging-multimessage pointer-lock web-share drag-and-drop pointerevents web-speech-api fullscreen-api reporting-api web-storage htmldialogelement-basic resize-event web-workers indexeddb-api resize-observer webgl-examples
And if I run
ls -A service-worker/, I get:
❯ ls -A service-worker/ simple-service-worker
And finally, running
ls -A service-worker/simple-service-worker/ shows:
❯ ls -A service-worker/simple-service-worker/ .prettierrc README.md image-list.js style.css CODE_OF_CONDUCT.md app.js index.html sw.js LICENSE gallery star-wars-logo.jpg
All that is left is to push to remote.
git push origin 146-chore-mo…dom-examples
NOTE: Do not squash merge this pull request, or else all commits will be squashed together as a single commit. Instead, you want to use a merge commit. You can read all the details about merge methods in their documentation on GitHub.
After you merge the pull request, go ahead and browse the commit history of the repo. You will find that the commit history is intact and merged. o/\o You can now go ahead and either delete or archive the old repository.
At this point having the remote configured for our target repo serve no purpose so, we can safe remove the remote.
git remote rm swtest
The steps to accomplish this task is then as follows:
# Clone the repository you want to merge git clone https://github.com/mdn/sw-test.git cd sw-test # Create your feature branch git switch -C prepare-repo-for-move # NOTE: With older versions of Git you can run: # git checkout -b prepare-repo-for-move # Create directories as needed. You may only need one, not two as # in the example below. mkdir service-worker mkdir service-worker/sw-test # Enable extendedglob so we can use negation # The command below is for modern versions of ZSH. See earlier # in the post for examples for bash and older versions of ZSH set -o extendedglob # Move everything except hidden files into your subdirectory, # also, exclude your target directories mv ^(sw-test|service-worker) service-worker/sw-test # Move any of the hidden files or folders you _do_ want # to move into the subdirectory mv .prettierrc service-worker # Add and commit your changes git add . git commit -m 'Moved all source files into new subdirectory' # Push your changes to GitHub git push origin prepare-repo-for-move # Head over to the repository on GitHub, open and merge your pull request # Back in the terminal, switch to your `main` branch git switch main # Delete your feature branch # This is not technically required, but I like to clean up after myself :) git branch -D prepare-repo-for-move # Pull the changes you just merged git pull origin main # Change to the root directory of your target repository # If you have not yet cloned your target repository, change # out of your current directory cd .. # Clone your target repository git clone https://github.com/mdn/dom-examples.git # Change directory cd dom-examples # Create a feature branch for the work git switch -C 146-chore-move-sw-test-into-dom-examples # Add your merge target as a remote git remote add -f ssw https://github.com/mdn/sw-test.git # Merge the merge target and allow unrelated history git merge swtest/gh-pages --allow-unrelated-histories # Add and commit your changes git add . git commit -m 'merging sw-test into dom-examples' # Push your changes to GitHub git push origin 146-chore-move-sw-test-into-dom-examples # Open the pull request, have it reviewed by a team member, and merge. # Do not squash merge this pull request, or else all commits will be # squashed together as a single commit. Instead, you want to use a merge commit. # Remove the remote for the merge target git remote rm swtest
Hopefully, you now know how to exclude subdirectories using the mv command, set and view shell configuration, and merge the file contents of a git repo into a new repository while preserving the entire commit history using only basic git commands.
About Schalk Neethling
I am a Mozillian, an evangelist, writer and developer with a passion for open source, web standards and accessibility. I have been so involved with these worlds that I feel they have become a part of me and cannot foresee a future where these topics will not be a part of my daily life.