Docker Eclipse Development Environment

Posted on  by admin

Running IntelliJ IDEA in Docker Container. Java developers have spoken and voted through developer surveys. Today’s favorite IDE is IntelliJ, taking anywhere between 60-80% of the developer base. Let’s take that to the next level and get you setup with Docker for your Java development environment with IntelliJ – community edition 2020.3. Our development platform is Ubuntu 18.04. Install in eclipse for development. Needed for a working development environment: MySQL database - we use a MariaDB docker container; Eclipse Java EE IDE for Web Developers. Local Apache tomcat for eclipse; Install MariaDB docker. Install and configure docker environment including docker-compose.

Docker is the defacto toolset for building modern applications and setting up a CI/CD pipeline – helping you build, ship and run your applications in containers on-prem and in the cloud.

Whether you’re running on simple compute instances such as AWS EC2 or Azure VMs or something a little more fancy like a hosted Kubernetes service like AWS EKS or Azure AKS, Docker’s toolset is your new BFF.

One aspect of Docker is to provide the same environment throughout the different stages of software development process: e.g. In development, staging and production. But does it really make sense.

But what about your local development environment? Setting up local dev environments can be frustrating to say the least.

Remember the last time you joined a new development team?

You needed to configure your local machine, install development tools, pull repositories, fight through out-of-date onboarding docs and READMEs, get everything running and working locally without knowing a thing about the code and it’s architecture. Oh and don’t forget about databases, caching layers and message queues. These are notoriously hard to set up and develop on locally.

I’ve never worked at a place where we didn’t expect at least a week or more of on-boarding for new developers.

So what are we to do? Well, there is no silver bullet and these things are hard to do (that’s why you get paid the big bucks) but with the help of Docker and it’s toolset, we can make things a whole lot easier.

In Part I of this tutorial we’ll walk through setting up a local development environment for a relatively complex application that uses React for it’s front end, Node and Express for a couple of micro-services and MongoDb for our datastore. We’ll use Docker to build our images and Docker Compose to make everything a whole lot easier.

If you have any questions, comments or just want to connect. You can reach me in our Community Slack or on twitter at @pmckee.

Let’s get started.

Prerequisites

To complete this tutorial, you will need:

  • Docker installed on your development machine. You can download and install Docker Desktop from the links below:
  • Sign-up for a Docker ID
  • Git installed on your development machine.
  • An IDE or text editor to use for editing files. I would recommend VSCode

Fork the Code Repository

The first thing we want to do is download the code to our local development machine. Let’s do this using the following git command:

git clone [email protected]:pmckeetx/memphis.git

Now that we have the code local, let’s take a look at the project structure. Open the code in your favorite IDE and expand the root level directories. You’ll see the following file structure.

├── docker-compose.yml
├── notes-service
│ ├── config
│ ├── node_modules
│ ├── nodemon.json
│ ├── package-lock.json
│ ├── package.json
│ └── server.js
├── reading-list-service
│ ├── config
│ ├── node_modules
│ ├── nodemon.json
│ ├── package-lock.json
│ ├── package.json
│ └── server.js
├── users-service
│ ├── Dockerfile
│ ├── config
│ ├── node_modules
│ ├── nodemon.json
│ ├── package-lock.json
│ ├── package.json
│ └── server.js
└── yoda-ui
├── README.md
├── node_modules
├── package.json
├── public
├── src
└── yarn.lock

The application is made up of a couple simple microservices and a front-end written in React.js. It uses MongoDB as it’s datastore.

Typically at this point, we would start a local version of MongoDB or start looking through the project to find out where our applications will be looking for MongoDB.

Then we would start each of our microservices independently and then finally start the UI and hope that the default configuration just works.

This can be very complicated and frustrating. Especially if our micro-services are using different versions of node.js and are configured differently.

So let’s walk through making this process easier by dockerizing our application and putting our database into a container.

Dockerizing Applications

Docker is a great way to provide consistent development environments. It will allow us to run each of our services and UI in a container. We’ll also set up things so that we can develop locally and start our dependencies with one docker command.

The first thing we want to do is dockerize each of our applications. Let’s start with the microservices because they are all written in node.js and we’ll be able to use the same Dockerfile.

Create Dockerfiles

Create a Dockerfile in the notes-services directory and add the following commands.

This is a very basic Dockerfile to use with node.js. If you are not familiar with the commands, you can start with our getting started guide. Also take a look at our reference documentation.

Building Docker Images

Now that we’ve created our Dockerfile, let’s build our image. Make sure you’re still located in the notes-services directory and run the following command:

docker build -t notes-service .

Now that we have our image built, let’s run it as a container and test that it’s working.

docker run --rm -p 8081:8081 --name notes notes-service

Looks like we have an issue connecting to the mongodb. Two things are broken at this point. We didn’t provide a connection string to the application. The second is that we do not have MongoDB running locally.

At this point we could provide a connection string to a shared instance of our database but we want to be able to manage our database locally and not have to worry about messing up our colleagues’ data they might be using to develop.

Local Database and Containers

Instead of downloading MongoDB, installing, configuring and then running the Mongo database service. We can use the Docker Official Image for MongoDB and run it in a container.

Before we run MongoDB in a container, we want to create a couple of volumes that Docker can manage to store our persistent data and configuration. I like to use the managed volumes that docker provides instead of using bind mounts. You can read all about volumes in our documentation.

Let’s create our volumes now. We’ll create one for the data and one for configuration of MongoDB.

docker volume create mongodb
docker volume create mongodb_config

Now we’ll create a network that our application and database will use to talk with each other. The network is called a user defined bridge network and gives us a nice DNS lookup service which we can use when creating our connection string.

docker network create mongodb

Now we can run MongoDB in a container and attach to the volumes and network we created above. Docker will pull the image from Hub and run it for you locally.

docker run -it --rm -d -v mongodb:/data/db -v mongodb_config:/data/configdb -p 27017:27017 --network mongodb --name mongodb mongo

Okay, now that we have a running mongodb, we also need to set a couple of environment variables so our application knows what port to listen on and what connection string to use to access the database. We’ll do this right in the docker run command.

docker run
-it --rm -d
--network mongodb
--name notes
-p 8081:8081
-e SERVER_PORT=8081
-e SERVER_PORT=8081
-e DATABASE_CONNECTIONSTRING=mongodb://mongodb:27017/yoda_notes notes-service

Let’s test that our application is connected to the database and is able to add a note.

curl --request POST
--url http://localhost:8081/services/m/notes
--header 'content-type: application/json'
--data '{
'name': 'this is a note',
'text': 'this is a note that I wanted to take while I was working on writing a blog post.',
'owner': 'peter'
}

You should receive the following json back from our service.

{'code':'success','payload':{'_id':'5efd0a1552cd422b59d4f994','name':'this is a note','text':'this is a note that I wanted to take while I was working on writing a blog post.','owner':'peter','createDate':'2020-07-01T22:11:33.256Z'}}

Conclusion

Awesome! We’ve completed the first steps in Dockerizing our local development environment for Node.js.

In Part II of the series, we’ll take a look at how we can use Docker Compose to simplify the process we just went through.

In the meantime, you can read more about networking, volumes and Dockerfile best practices with the below links:

MassBank needs a MariaDB database running somewhere, a tomcat server for the webapp and the maven build system for installation. This can be provided directly by the hosting system or in docker containers. Although in principle it might be possible to run this webapp on a windows server, we haven’t tested it and it will not run out of the box.

Besides running MassBank on a server system it is also possible to run it in the integrated tomcat server within eclipse for easy development and debugging. Our development platform is Ubuntu 18.04.

Install in eclipse for development

Needed for a working development environment:

  1. MySQL database - we use a MariaDB docker container
  2. Eclipse Java EE IDE for Web Developers.
  3. Local Apache tomcat for eclipse

Install MariaDB docker

Install and configure docker environment including docker-compose. Make sure that your user is in the group ‘docker’ and adjust your root password for the database in compose/mariadb-docker-compose.yml.

Start the database with docker-compose -f compose/mariadb-docker-compose.yml up -d and check with docker ps for massbank_mariadb. Check database connectivity with mysql -u root -h 127.0.0.1 -p.

Install massbank.conf

Adjust settings in massbank.conf and copy it to /etc.

Import project into Eclipse

Download and install the Eclipse IDE for Java EE Developers. Start eclipse and import the folder ‘MassBank-Project’ from this repo: File -> Import -> Existing Maven Project -> Select this repo for import. Download Apache Tomcat and extract it to your preferred folder.Then create a Tomcat server in eclipse.

Populate database

Before the MassBank webapp can serve the data the database needs to be filled.

First otion is to clone the data from MassBank-data to a local directory and make sure the ‘DataRootPath’ in ‘massbank.conf’ is correctly set to your data repo. Run ‘/MassBank-lib/src/main/java/massbank/RefreshDatabase.java’ from to populate your database. This way a dev-snapshot of the data can be installed.

Second option is to download the sql dump from the latest release and import it into the database with mysql -u root -h 127.0.0.1 -p < MassBank-2020.10.sql.

Run MassBank webapp

Run the MassBank-project on the Tomcat server and access MassBank at http://localhost:8080/MassBank/.

Install as server system with docker/multiple instances on one server possible

A docker compose file named compose/full-service.yaml is provided to install all required container to run MassBank as a server system. The only requirement is the docker environment including docker-compose. Make sure that your user is in the group ‘docker’. A setup script called install.sh is provided to make deployment easy.

Install one instance with docker-compose

To set up the MassBank webapp the source of the webapp and the data repository are needed. Get them with:

If you want to use the install.sh-script from MassBank-web both repositories need to be in the same directory. Usage is:

install.sh supports four operations:

  • start a MassBank service (incl. the required database)
  • stop and remove a MassBank service
  • refresh the content of the MassBank
  • deploy the code to the tomcat container
  • instance is used to separate the networks of different MassBank servers and determines the port of the service, 0 serves at 8080, 1 serves at 8081 …

Several networks with independent instances of MassBank are possible with this deployment method. To deploy to standard port 8080 use ./install.sh start 0. After 30 min you can find a MassBank instance at http://<your-ip>:8080/MassBank. To upload new content update the data repo (needs to be in ../MassBank-data) and issue ./install.sh refresh 0. To remove this MassBank use ./install.sh stop 0.

Install multiple instance with docker-compose

The deployment uses the codebase in MassBank-web and the data in MassBank-data in the same parent directory for one deployment. Because the data in MassBank-data is not only used for deployment, but also for serving it is recommended to have a separate data repo for each instance of MassBank, e.g.:

With this layout its easy to have several instances with different codebase / data on the same server. To start MassBank1 go to MassBank1/MassBank-web and use ./install.sh start 1. Your server will be at http://<your-ip>:8081/MassBank. To start MassBank2 go to MassBank2/MassBank-web and use ./install.sh start 2. Your server will be at http://<your-ip>:8082/MassBank. The second parameter of the install script is used to separate the different instances by using different subnets and ports on the local machine.

Docker Dev Environment

Install as server system with Vagrant

A Vagrantfile is provided for easy installation of a MassBank-server. This config creates a Ubuntu VM with IP 192.168.35.18. Inside this VM the docker-compose mechanism as described above is used to create a MassBank-server on port 8080. Additionally a Apache2 http server is installed as reverse proxy. The config can be found in conf/apache2. Please modify if needed. The final MassBank site will be available at https://192.168.35.18/MassBank. The installation uses the MassBank-data repository from ../MassBank-data. You can modify the location in the Vagrantfile. The installation can be started with vagrant up.

Sitemaps, robots and scrapers

MassBank will serve a sitemap index which points to the actual sitemaps containing links to the individual sitempas with a maximum of 50k entries each. These sitemaps can be used by search engines or for semantic data extraction.

PIWIK log analyser (https://piwik.org/)

The default MassBank server installation includes the PIWIK log analyser. Consider that user tracking has privacy issues.The default preset follows very strict rules according to http://piwik.org/docs/privacy/ and only the following usage data:

  • Site from which the file was requested
  • Name of the file
  • Date and time of the request
  • Volume of data transferred
  • Access status (file transferred, file not found)
  • Description of the type of web browser used
  • IP address of the requesting computer shortened by the last six digits.

The creation of personal user profiles is therefore precluded.

customised piwik tracking code

The tracking code for piwik is specific for each site and configuration in order to fullfil your local legal and personal requirements!The tracking code for http://your_server_url/MassBank is called from /var/www/html/MassBank/script/Piwik.js.Make sure to replace the default code with your customised tracking code. Exclude the script tags.Make also sure that you customise your superuser if using the default bootstrap.sh!

Main branches

We use two main branches, main and dev. All development should happen in dev and we define every commit to main to be a release. When the source code in the dev branch reaches a stable point and is ready to be released, all of the changes should be merged back into main somehow and then tagged with a release number. How this is done in detail will be discussed further on. To use all of the command lines below the github/hub tool is required.

Supporting branches

The different types of branches we may use are:

  • Feature branches
  • Release branches
  • Hotfix branches

Feature branches

Branch off from: dev

Must merge back into: dev

Branch naming: anything except main, dev, release-* or hotfix-*

Feature branches are used to develop new features.

Docker Eclipse Development Environment

Creating a feature branch

Incorporating a finished feature on dev

Release branches

Docker Visual Studio Development Environment

Branch off from: dev

Must merge back into: dev and main

Branch naming: release-*

Release branches support preparation of a new production release. They allow for minor bug fixes and preparing the version number for a release. It is exactly at the start of a release branch that the upcoming release gets assigned a version number.

Creating a release branch

Finishing a release branch

When the state of the release branch is ready to become a real release, the release branch is merged into main with a pull request and tagged for easy future reference.

Wait for all checks to finish. There are most likely conflicts, which need to be resolved first.

Docker

Resolve conflicts, commit and push.

If this is done, the release can be merged to main.

If there were any changes in the release branch we need to merge them back to dev.

This may well lead to merge conflicts, which needs to be fixed. If so, fix it and commit.

Now we are done and the release branch may be removed.

Hotfix branches

Branch off from: main

Must merge back into: dev and main

Branch naming: hotfix-*

Hotfix branches are very much like release branches in that they are also meant to prepare for a new production release. They arise from the necessity to act immediately upon an undesired state of a live production version.

Docker For Local Development Environment

Creating a hotfix branch

Then, fix the bug and commit the fix in one or more separate commits.

Finishing a hotfix branch

When finished, the bugfix needs to be merged back into main, but also needs to be merged back into dev.First, update main and tag the release.

Wait for all checks to finish. Now the release can be merged to main.

Next, include the bugfix in dev, too:

The one exception to the rule here is that, when a release branch currently exists, the hotfix changes need to be merged into that release branch as well.Finally, remove the temporary branch:

Docker Development Tools

The release strategy of MassBank-data is similar to the one of MassBank-web and is described below. Additionally it needs to be taken care of the validation and the codebase for the validation.

Main branches

We use two main branches, main and dev. All development should happen in dev and we define every commit to main to be a release. When new data in the dev branch has reached a stable point and is ready to be released, all of the changes should be merged back into main somehow and then tagged with a release number. The data of main should be validated with the main of MassBank-web and the data of dev should be validated with dev of MassBank-web. If the data contains incompatible changes MassBank-web needs to be released first, because before a merge to main is possible a validation needs to be successfully. The code base for the validation is defined in travis.yml. Depending on the given version the bump-version.sh script adjusts the content of travis.yml to choose the corresponding branch from MassBank-web. If the version tag contains SNAPSHOT like in 2019.09-SNAPSHOT its identified as dev data branch and travis will pull the dev branch of MassBank-web for validation. If SNAPSHOT is missing its identified to be a release version and travis will pull the main branch of MassBank-web for validation. All versions which get a release tag in github are used by a webhook from zenodo and get a DOI attached. The authors list of the record at zenodo needs to be manually edited to contain MassBank-consortium and its contributors.

The release process is discussed in detail below. To use all of the command lines below the github/hub tool is required.

Release branches

Release branches are created in preparation of a release.

Branch off from: dev

Must merge back into: main

Branch naming: release-*

Release branches support preparation of a new production release. They allow for preparing the version number for a release.

Creating a release branch

Finishing a release branch

When the state of the release branch is ready to become a real release, the release branch is merged into main with a pull request and tagged for easy future reference.

Wait for all checks to finish. If all tests are successfull the release can be merged to main and tagged as release.

Resolve conflicts and commit.

Docker Java Development Environment Eclipse

Now we are done and the release branch may be removed

Docker Image For Development Environment

and the version of the dev branch should be incremented.