What is DacaCacheStorage?

DacaCacheStorage is a library for creating a key-value store database. The key is simply a string that must be unique. The value linked to the key is generic.
 >> DataCacheStorage offers great flexibility of use.
 >> It is very light to set up.
 >> It also allows you to manage the expiration time of your data.
 >> It allows you to transport stored items from one server to another.
 >> It also allows you to reload the items in memory after a restart or a crash of your application.

Integrating DataCacheStorage into your Spring Boot project

+

To simplify the reading of this article, we will name the DataCacheStorage library « Dacas ».

We will try to explain its integration into your project as clearly as possible.

We will show you how to integrate this solution into your Spring Boot application using Maven.

Prerequisite

The prerequisites requested are:
 – Good knowledge of the java language.
 – Basic knowledge of SpringBoot Framework.
 – Dependency management using Maven or Gradle.

In this demonstration, we will use:
 – at least version 8 of Java.
 – the IntelliJ IDE
 – Spring Boot in version 2.2.5.RELEASE
 – Maven
 – DataCacheStorage in version 1.0

Note that you can retrieve this sample project for this demonstration through this Github link or at the bottom of this page.

To follow this tutorial, you will need a Spring Boot application.
You can easily generate a Spring Boot application with Spring Initializr. To do so, go to https://start.spring.io/

 – Generate an application blank of all other dependencies if you do not already have an application.

Your project will have the following structure.

Adding DataCacheStorage to your project

In this example we will use the first version of DataCacheStorage.

To see the available versions, go to the Maven central office:

https://mvnrepository.com/artifact/com.digi-lion/datacachestorage

In the pom.xml file, add the following dependency:

<dependency>
          <groupId>com.digi-lion</groupId>
          <artifactId>datacachestorage</artifactId>
          <version>1.0</version>
</dependency>

This is what your pom.xml file will now look like.

Using your IDE or command line, run the Maven update command.

 >> Command line:
mvn clean install -U

>> Via Eclipse:
Right click on your project, then Maven > Update Snapshots.

>> Via IntelliJ:
Settings Menu -> Maven -> Import Maven projects automatically.

You will now have integrated the Dacas library, you should also know that you will also integrate the libraries :

-> Jackson-core (version: 2.10.3)
-> Jackson-annotations (version : 2.10.3)
-> Jackson-databind (version : 2.10.3)
-> Jackson-datatype-jsr310 (version : 2.10.3)

Since Dacas uses JSON as a backup format for your data, the Jackson library is used for this one.

Creating the DataCacheStorage configuration

For our minimalist example, we have created:
-> a Car object (our POJO)
-> a DemoService class (our service)
-> a DataCacheStorageConfig class (our DataCacheStorage configuration class)
-> a « dataCacheStorageFile » directory in the resources folder (the folder where our backup file is stored)

First, we’re going to set up the Dacas configuration.
I invite you to copy this class. We will explain each instruction of this one as we go along.

package com.example.demo.configuration;

import com.digilion.dataAccessClass.enumerations.DacaStorageSupportEnum;
import com.digilion.dataAccessClass.manager.DacasManager;
import com.digilion.dataAccessClass.transaction.DacasTransaction;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.Scheduled;

/**
* Configuration of DataCacheStorage<br>
* <p>
* <div>
* Injection of DataCacheStorage and Build manager. Purge and storage support
* </div>
* </p>
*
* @author Digital Lion
* @since 1.0
*/
@Configuration
public class DataCacheStorageConfig {

/**
* Definition of logger
*/
Logger logger = LoggerFactory.getLogger(DataCacheStorageConfig.class);

/**
* Default constructor
*/
public DataCacheStorageConfig() {
}

@Value("${dacas.file.path}")
private String jsonFilePath;

@Value("${dacas.max.object.in.memory}")
private int limit;

@Value("${dacas.max.time.to.live}")
private Long ttl;

/**
* Run Dacas manager
*/
@Bean
public DacasManager runDacas() {
DacasManager dacasManager = new DacasManager(DacaStorageSupportEnum.FILE, limit, jsonFilePath, ttl);
logger.info("Initializing Dacas.");
return dacasManager;
}

/**
* Schedule task for automatically purge expired elements on memory
*/
@Scheduled(initialDelay = 1000*30, fixedDelay = 1000*60*30)
private void purge() {
DacasTransaction.purge();
}

/**
* Schedule task for automatically copying elements from memory to backup file
*/
@Scheduled(initialDelay = 1000*60, fixedDelay = 1000*60*60)
private void autoPush(){
DacasTransaction.push();
}
}

Let’s now detail this DataCacheStorageConfig class.

This class is annotated @Configuration, it can be started using the component analysis (Dependency Injection).

Let’s define a logger that we will use later. Feel free to use your favorite logger, in this example we used slf4j.

Let’s move on to more interesting things:

We need to define 3 properties:
  -> the path to our backup file (jsonFilePath).
  -> the maximum number of elements we can inject in Dacas (limit).
  -> the default expiration date (in seconds) if an element injected in Dacas has no clearly defined expiration date (ttl) .

So we will have to define the three following keys in our property file. We’ll come back to this later.

Let’s now declare a bean thanks to the @Bean decorator. This is what we do with the runDacas() method.

The DacasManager constructor takes 3 parameters :
  –> DacaStorageSupportEnum.
             —> FILE (storage in memory and saving memory to file)
             —> MEMORY (storage only in memory)
  –> limit (defined above)
  –> jsonFilePath (defined above)
  –> ttl (defined above)

The last two methods of classification (not mandatory but highly recommended) allow to:
 –> automatically purge items whose expiration date has passed.
 –> create a backup of the data present in memory to a backup file.

These two scheduled tasks are annotated @Scheduled. In our example, these tasks will be executed every half hour for the purge and every hour for the backup.

It is up to you to configure the frequency of these tasks according to the needs of your applications.


Please note that every time you make a backup, Dacas deletes the expired data beforehand.

To finish with the configuration, you must add in your .properties file (in our example: application.properties) the following three keys:

dacas.file.path=/.../demo-dacas/src/main/resources/dataCacheStorageFile/
dacas.max.object.in.memory=10000001
dacas.max.time.to.live=86400

The key « dacas.file.path » defines the path to the backup file (the file will have the name: « dacasmemoriesvalues.dacas »).
The key « dacas.max.object.in.memory » defines the maximum number of elements allowed in memory.
The key « dacas.max.time.to.live » defines the default maximum lifetime of an element in memory. This default value is only applied to elements packed in a DacasElement*, injected in Dacas, and without a defined expiration date. The value is in seconds.

*We will explain what a DacasElement is in the next paragraph.

Storage of elements

DacasElement allows you to package your object to be injected in memory. Thanks to DacasElement, each of your elements will have a clearly defined expiration date. If you don’t define it, the default date defined in the Dacas configuration will be assigned to it.

You can however inject an object directly into Dacas, but this is a bad practice because you don’t define an expiration date for it.

However, an expiration date will be set for it. This date will be the one entered in the Dacas configuration (ttl). The element will be reloaded by the application when it is restarted as long as the expiration date of the element is not exceeded. The only « clean » way to remove these elements before their expiration date is to use the cleanup() method. However, all the cache will be emptied indiscriminately, expired or not.
However, if you have set Dacas to memory-only mode (DacaStorageSupportEnum.MEMORY), your item will remain in memory as long as your application is started.

For all manipulations of your data in Dacas simply use DacasTransaction.
DacasTransaction gives you the possibility to add, read, purge, dump, backup your data. You do not need DacasTransaction for all your operations.

For example let’s create 3 examples of methods to inject an element in Dacas.
The first exampleInjectionObjectDataIntoDacas() without using DacasElement. (Bad practice)
The second exampleInjectionDacasElementIntoDacas() using DacasElement. (Good practice for injecting elements into Dacas)
The last exampleInjectionDacasElementIntoDacasWithBuilder() using DacasElementBuilder. (Good practice for injecting elements into Dacas)

package com.example.demo.services;

import com.digilion.dataAccessClass.element.DacasElement;
import com.digilion.dataAccessClass.transaction.DacasTransaction;
import com.example.demo.bo.Car;
import org.springframework.stereotype.Service;

import java.time.LocalDateTime;

/**
* Example of service class
*
* @author Digital Lion
* @since 1.0
*/
@Service
public class DemoService {

/**
* Default Constructor
*/
public DemoService() {
}

/**
* Example of injecting Object into DataCacheStorage.
* <p>
* <u>Inserting Objects in Dacas without packaging them into a DacasElement is not recommended</u>
* </p>
* @param carToInject
*/
public void exampleInjectionObjectDataIntoDacas(Car carToInject) {
// Committing into DacaStorage
// Since you didn't define DateTimeStorage and/or ExpiredDateTime, the object will be converted to DacasElement with the default expiring date defined in configuration.
DacasTransaction.commit(carToInject.getLicensePlate(), carToInject);
}

/**
* Example of injecting Object packaged into DacasElement to DataCacheStorage.
* @param carToInject
*/
public void exampleInjectionDacasElementIntoDacas(Car carToInject) {
DacasElement dacasElement = new DacasElement();
dacasElement.setValue(carToInject);
// It is recommended to fix DateTimeStorage and ExpiredDateTime for your DacasElement.
// If you don't define DateTimeStorage and/or ExpiredDateTime, default values defined on configuraton are set.
LocalDateTime now = LocalDateTime.now();
dacasElement.setDateTimeStorage(now);
dacasElement.setExpiredDateTime(now.plusHours(2));

// Committing your object packaged into DacasElement in DacasStrorage
DacasTransaction.commit(carToInject.getLicensePlate(), dacasElement);
}

/**
* Example to inject Object packaged into DacasElement (using Builder) to DataCacheStorage.
* @param carToInject
*/
public void exampleInjectionDacasElementIntoDacasWithBuilder(Car carToInject) {
// You can also use Builder
DacasElement dacasElementBuilder;
LocalDateTime now = LocalDateTime.now();
dacasElementBuilder = new DacasElement.DacasElementBuilder<>().value(carToInject).dateTimeStorage(now).lifeTime(now.plusHours(2)).build();

// Committing your object packaged into DacasElement in DacasStrorage
DacasTransaction.commit(carToInject.getLicensePlate(), dacasElementBuilder);
}

/**
* Example of accessing your data
* @param key
* @return car
*/
public Car exampleAccessToYourData(String key) {
// When you retrieve your value, you directly retrieve your value and not a DacasElement.
// Cast your retrieved data! And that's all.
return (Car) DacasTransaction.get(key);
}

/**
* Example of pushing element in memory into the dacasmemoriesvalues.dacas file
*/
public void forceBackup() {
// If you configure storage support as a file,
// method push() send all data from memory to file referenced into properties file.
// Before writing, all expired data will be flushed.
DacasTransaction.push();

}

}

DacasManager.dacasStrorage gives you access to some utility methods to manipulate your elements injected into Dacas.

Element recovery

From our main class, we simply created three myBeetle, myFiat and myFord Car objects.
We injected these three elements into Dacas.

The key we chose for our elements must, of course, be unique and of String type. For our example, we chose the license plate of our vehicles.

Then we forced the writing of our information in memory to the backup file. This step is not necessary, we force this writing during this example in order to describe the created backup file. Dacas will take care of the backup itself thanks to the scheduled task described above in this article.

Finally, we will retrieve the data using its unique key.
Since the object recovered from Dacas is of type Object, you have to proceed to a cast, here we cast in type Car.

package com.example.demo;

import com.example.demo.bo.Car;
import com.example.demo.services.DemoService;
import com.example.demo.services.PerformanceReducedTest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

import java.time.LocalDate;
import java.time.Month;

@SpringBootApplication
public class DemoApplication {

/**
* Definition of logger
*/
private static Logger logger = LoggerFactory.getLogger(DemoApplication.class);

public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
runDataCacheStorageDemo();
}

/**
* Demonstration method
*/
public static void runDataCacheStorageDemo() {
DemoService demoService = new DemoService();
// Example of object to inject into DataCacheStorage
Car myBeetle = new Car("VolksWagen", "Beetle", 50, LocalDate.of(1972, Month.JANUARY, 12), "MB-123-RC");
Car myFiat= new Car("Fiat", "500", 38, LocalDate.of(1982, Month.AUGUST, 25), "TR-987-KV");
Car myFord= new Car("Ford", "GT", 550, LocalDate.of(1991, Month.NOVEMBER, 03), "GT-987-LM");

// Example of direct injection of Object into DataCacheStorage.
// Possible but not recommended because you don't set an expiration date at your data
demoService.exampleInjectionDacasElementIntoDacas(myBeetle);

// Example of good practice for the injection of Object into DataCacheStorage.
// Your object is packaged into DacasElement.
demoService.exampleInjectionObjectDataIntoDacas(myFiat);
demoService.exampleInjectionDacasElementIntoDacasWithBuilder(myFord);

// We are forcing the backup before scheduled task execution (for this example of application)
demoService.forceBackup();

// Example of accessing your data
Car myBeetleRetrieved = demoService.exampleAccessToYourData("MB-123-RC");
logger.info("First Circulation date of my retrieved car : " + myBeetleRetrieved.getFistCirculationDate());
}
}

If you have items that require immediate writing to the backup file after being injected into Dacas; use the commitAndPush() method. This method immediately writes the elements in memory to the backup file. Your backup file will then be up to date.

Generated backup file

If you choose a File backup type (DacaStorageSupportEnum.FILE), your elements will be both kept in memory and also saved as JSON in a file named: dacasmemoriesvalues.dacas . We have previously configured the path to this one in Spring’s properties (application.properties).

Here is the content of this file for our example.

Restart or crash of the application

When setting up the DacasManager configuration, Dacas checks whether a dacasmemoriesvalues.dacas file exists in the directory specified in the properties of your application.

If this file does not exist, DacasManager will generate a new one. Make sure you have write permission for your server to write to this directory.

If the file exists, DacasManager will browse through the file, items with an expiration date that has passed are deleted from the file. All other items are reloaded into memory.

If your application crashes for some reason, when it is restarted, DacasManager will reload the elements present in the backup file. Please make sure that you configure the correct backup frequency for your data.

Depending on the context of your application, you can also use the Dacas backup file to share or transfer your data from one server to another.

Some figures

Element injection time in DataCacheStorage.

Reading time of the values present in DataCacheStorage.
We can guarantee that you will always have access to your data in less than a millisecond.

In blue, the time to inject elements into Dacas, then make a backup once the injection is finished.
In red, the time to inject and save elements one by one, then make the next element injection.

Time to inject and save elements one by one. For applications with very large amounts of data to be added in memory in a very recurrent way (more than 10,000 elements per second), it is strongly recommended not to make a backup after each injection of your data in memory. Prefer backup by batch ( DacasTransaction.commitMultiAndPush( ) ) or by time interval.

DataCacheStorage is nearly 400 times faster by optimizing the intervals between each backup.
This curve shows the importance of a good configuration of DataCacheStorage’s automated tasks (backup and purge) to optimize Dacas performance and improve the performance of your application.

Time required for DataCacheStorage to browse through all items to purge expired items and create an up-to-date backup.

The previous tests were carried out on seven machines of different configurations. The values are an average of the seven test results.

Useful links

Thank you for taking this tutorial. I hope I made it as clear as possible. Do not hesitate to share with me your feelings, points of improvement or blockage through the contact form, by clicking here.

See you soon, have fun!