Solution for controlling when you update large files: submodules (Git & Dev Tools)

Why This Matters

In large projects, it's common to have multiple repositories that contain related files or dependencies. Managing these dependencies manually can be tedious and error-prone. Git submodules provide a solution for handling such situations by allowing you to include one repository as a subdirectory of another repository. This lesson will guide you through the process of creating, updating, and managing Git submodules.

By using Git submodules, developers can maintain separate repositories for different components while keeping them synchronized within the main project. This promotes better organization, easier collaboration, and less duplication of code.

Prerequisites

To follow this tutorial, you should have a basic understanding of:

1. Git fundamentals (committing, branching, merging)
2. Navigating the command line
3. Familiarity with version control systems
4. Understanding of how to clone and fork repositories on platforms like GitHub
5. Basic knowledge of Linux/Unix-like operating systems (e.g., navigating directories, creating files)
6. Knowledge of common file structures and directory layouts in software projects
7. Familiarity with the concept of dependencies in software development

Core Concept

Creating a Submodule

To create a submodule in an existing repository, navigate to the parent directory and use the following command:

git submodule add <url> <path> --init

Replace ` with the URL of the repository you want to add as a submodule, and with the desired path for the submodule within the parent repository. The --init` flag initializes the new submodule.

After running this command, Git will clone the specified repository into the given path and create a new entry in the .gitmodules file.

Example: Adding a submodule called my-library from GitHub

cd my-project
git submodule add https://github.com/exampleuser/my-library libraries/my-library --init

Updating a Submodule

To update an existing submodule, navigate to the submodule directory and run:

git pull

If you want to update all submodules at once, navigate to the parent repository directory (e.g., my-project) and use:

git submodule update --init --recursive

This command will fetch the latest version of each submodule and initialize them if necessary.

Example: Updating a submodule called my-library

1. Navigate to the submodule directory (libraries/my-library):

cd libraries/my-library

1. Run git pull to fetch the latest version of the submodule:

git pull

1. To update all submodules at once, navigate back to the parent repository directory (my-project) and run:

git submodule update --init --recursive

Removing a Submodule

To remove a submodule, first remove the corresponding entry from the .gitmodules file:

git rm <path>

Then, commit the changes:

git commit -m "Remove submodule"

Finally, run:

git submodule sync && git submodule update --recursive --cleanup --prune

This command will remove the submodule from the repository and clean up any unused data.

Example: Removing a submodule called my-library

1. Remove the entry for my-library from the .gitmodules file:

git rm libraries/my-library

1. Commit the changes:

git commit -m "Remove submodule"

1. Run the cleanup command:

git submodule sync && git submodule update --recursive --cleanup --prune

Tracking Submodule Changes

By default, Git does not track changes made to the files within a submodule. To start tracking changes, navigate to the submodule directory and run:

git init && git add .

Then, commit the changes in the parent repository:

cd ..
git commit -am "Track changes to submodule"

Example: Tracking changes in a submodule called my-library

1. Navigate to the submodule directory (libraries/my-library):

cd libraries/my-library

1. Initialize a new Git repository and add all files:

git init && git add .

1. Navigate back to the parent directory (my-project) and commit the changes:

cd ..
git commit -am "Track changes to my-library"

Submodules vs. Merge Strategies

Submodules are different from merge strategies like recursive, which handle merges between repositories at the file level instead of the directory level. While merge strategies can be useful for managing conflicts within a single repository, submodules are better suited for managing dependencies across multiple repositories.

Worked Example

Let's say you have a parent repository named my-project, and you want to include a submodule called my-library. To do this, follow these steps:

1. Navigate to your my-project directory:

cd my-project

1. Add the my-library repository as a submodule:

git submodule add https://github.com/exampleuser/my-library libraries/my-library --init

1. Verify that the .gitmodules file has been updated:

cat .gitmodules

1. To update the my-library submodule, navigate to the submodule directory and run:

cd libraries/my-library
git pull

1. To update all submodules at once, navigate back to the parent repository directory (my-project) and run:

git submodule update --init --recursive

1. If you want to start tracking changes in my-library, navigate to the submodule directory and run:

cd libraries/my-library
git init && git add .
cd ..
git commit -am "Track changes to my-library"

Common Mistakes

1. Forgetting to Initialize a New Submodule

When creating a new submodule, make sure to initialize it with the --init flag:

git submodule add <url> <path> --init

2. Not Committing Submodule Changes

Changes to submodules are not automatically tracked by Git. Always commit changes to both the parent repository and the submodule directory before pushing your code.

3. Using Relative URLs for Submodules

When adding a submodule, use the full URL instead of a relative path. Relative paths may cause issues when cloning the repository or working with multiple branches.

4. Not Handling Submodule Merges Correctly

When merging branches in the parent repository, make sure to run git submodule update --init --recursive to fetch the latest versions of all submodules and resolve any conflicts that may arise.

###5. Ignoring Submodules in Gitignore

It's important not to ignore submodules in your .gitignore file, as this can prevent them from being tracked by Git. If you want to exclude specific files within a submodule, create a separate .gitignore file inside the submodule directory instead.

###6. Not Updating Submodules Automatically on Clone

By default, Git does not clone submodules automatically when cloning a repository. To enable this behavior, add the following line to your .git/config file:

core.submodule = true

###7. Not Handling Submodule Updates in CI/CD Pipelines

When using continuous integration and deployment (CI/CD) tools like Jenkins, Travis CI, or GitHub Actions, make sure to include steps for updating submodules during the build process to ensure that the latest versions are always used.

Practice Questions

1. How can you create a Git submodule named my-dependency in your current directory, using the URL https://github.com/exampleuser/my-dependency?

git submodule add https://github.com/exampleuser/my-dependency my-dependency --init

1. What command would you use to update all submodules in your repository at once?

git submodule update --init --recursive

1. How can you remove the my-dependency submodule from your repository?

First, remove the entry from the .gitmodules file:

git rm my-dependency

Then, commit the changes and clean up unused data:

git commit -m "Remove submodule"
git submodule sync && git submodule update --recursive --cleanup --prune

1. How can you start tracking changes to a submodule named my-dependency?

Navigate to the submodule directory and run:

cd libraries/my-dependency
git init && git add .
cd ..
git commit -am "Track changes to my-dependency"

FAQ

Q1. Can I have multiple versions of a submodule in my repository?

A1. Yes, you can have multiple versions of a submodule by creating separate branches for each version and adding them as different submodules with unique paths.

Q2. What happens if I delete the .gitmodules file?

A2. If you delete the .gitmodules file, Git will no longer recognize the submodules in your repository. You'll need to recreate the submodule entries manually or clone the repository again to restore them.

Q3. Can I push a repository with submodules directly to GitHub?

A3. No, you cannot push a repository with submodules directly to GitHub. Instead, you should use Git's git push --recurse-submodules=on-demand command or configure GitHub to handle submodules automatically.

Q4. How can I handle submodule updates in my CI/CD pipeline?

A4. To handle submodule updates in your CI/CD pipeline, include a step that runs git submodule update --init --recursive before any build or deployment steps. This ensures that the latest versions of all submodules are used during the build process.

Q5. What is the difference between Git submodules and git subtrees?

A5. Git submodules and git subtrees are both methods for including external repositories within a project, but they have some key differences. Submodules create separate repositories within your main repository, while subtrees merge the contents of an external repository directly into your main repository's history. Submodules are more suitable for managing dependencies between multiple repositories, while subtrees may be better suited for merging code from a single repository into another.