Poll ftp server to pick up shipment files
Description
Attachments
Activity
Thanks @Josh Zamor
A bit more background on the diagram may help.
In OpenLMIS, we expect that there may be multiple ftp locations that we need to monitor for shipment files. The transfer properties api also supports persisting multiple shipment transfer properties.
The main idea is to use Spring Integration's Message Channel to implement the integration.A channel adapter is an endpoint that connect the channel to a form of transport. In our case, we would need a message adapter to connect to the ftp servers.
In an ideal world, we may want to have multiple producers sending messages to one message channel.
Spring integration (before v5.0.7) does not support monitoring multiple servers. In version 4.x of spring integration that we can outright use, the work around that I was able to find for this problem is to create multiple channels (one for each ftp server we may want to monitor) That is what I was trying to depict in the diagram.
Other than this subtle design decision, The component used in the diagram are mostly coming from the spring-integration documentation. The two links I provided above would describe them.
I did not check the efforts required to upgrade to spring boot 2. If that multiple server support was available with a spring boot upgrade, the diagram above would not hold. I will reach out to you on slack about the spring boot upgrade early next week.
Very roughly; at a very much mixed level of details, the following sequence diagram says a lot of what the flow would look like.
Looks great @Elias Muluneh.
What I'm seeing is the use of Spring Integration (great) and I also saw the note about Spring 5 / Spring Boot upgrade.
A couple questions:
I've been tracking a Spring Boot 1.x to 2.x upgrade in Reference Data (as in I have a private branch where I've been feeling out the problem areas). I think you're right about taking the upgrade out of this ticket, however I'm wondering if we can squeeze something in here for it. Perhaps we could create a fulfillment framework upgrade branch and see if it's easier than reference data (it's smaller at the very least). Or maybe it's a smaller bite of that apple - anything that gets non-trivial upgrades done is a win I think. Perhaps ping me on Slack?
I didn't understand the diagram, or much of the flow described, in great depth. That might be alright if you're solid on what the logic is. If you're looking for feedback on that part perhaps a sequence diagram might help (https://www.websequencediagrams.com/).
@Josh Zamor @Wesley Brown FYI: I am planning to implement the polling for shipment file as documented in this ticket.
I did not find similar polling/integration implementation in OpenLMIS so I wanted to make sure you are aware of this. Please let me know your thoughts or if you have any concerns.
This ticket is also related to OLMIS-5671, OLMIS-5672, OLMIS-5673, OLMIS-5674 and OLMIS-5675. Looking at these other tickets may provide a bit more detail about the general direction that shipment file processing is being implemented. Some of the tickets like the ability to update the connection credential at runtime may be ‘good to have’ and will not make it in the current sprint.
Details
Details
Assignee
Reporter
Story Points
Components
Sprint
Affects versions
Priority
Major
Time Assistant
Open Time Assistant
Time Assistant
When the fulfillment service starts, Create necessary spring-integration beans that will poll the FTP server/s for shipment files.
The configuration of these beans may look as follows.
Acceptance Criteria:
The FTP credentials for the shipment files are read from transfer_properties table.
The polling should only pick up files ending with .csv
If any file is found in the server, it should be copied to the local directory specified in the transfer_properties table. The remote file should be deleted.
Remote directory provided in the transfer properties should be considered the base remote path. the polling should try to read files from [remote_path]/incoming directory.
The spring integration beans should be created only for "FTP" SHIPMENT transfer types.
The Polling interval should be read from the environment variable.
The configurations should apply for each warehouse that has a shipment ftp configuration,
NOTE: The fulfillment service uses spring 4.x. There are more advanced features starting from spring integration 5.0.7 to support Polling of multiple FTP servers and directories. It may not be possible to use Spring Integration 5.0.7 in the fulfillment service without first upgrading the spring/spring boot version of fulfillment service. That is out of scope for this ticket. The plan therefore is to implement the integration using a spring integration version that matches the current version of spring.