Note: This applies only to CIM sockets using XML / CSV metadata files
This does not apply to the simple CIM sockets using Flat Files (without metadata)
Best practices for configuring CIM and sending files to the watch folder:
Note: The example listed below uses csv. However, the process is the same for xml files.
- Upload the transform file first before configuring the socket.
- Copy the files (PDFs) to the watch folder.
- Copy the csv metadata last. Here is an example of a best practice when copying the .csv to the watch folder:
- Copy csv to the watch folder by temporarily giving it a non-csv extension - eg. metadata.csv.tmp. This prevents the cim from processing the csv while it is still in transit to the watch folder.
- Rename metadata.csv.tmp to metadata.csv to start triggering the ingestion.
Other things to consider:
- Files uploaded in different batches should not have same file name to avoid overwriting in the watch folder. Always have unique names for files you ingest so that you know which batch they are from. A good strategy would be to append the batch name to the files, also ensuring that the batch name is unique. Alternatively, add a time stamp to the metadata file name to ensure uniqueness.
- Keep batches to a "manageable" size - eg. 500-1000 files per csv (batch). The bigger the batch, the more things to debug if something goes wrong.
- Look at the error directory to find more info about ingestion errors.
- Clean the contents of the error directory from time to time to free up disk space.