{"id":1040,"date":"2025-06-22T13:46:46","date_gmt":"2025-06-22T13:46:46","guid":{"rendered":"https:\/\/opturo.com\/staging\/?page_id=1040"},"modified":"2025-07-03T14:22:54","modified_gmt":"2025-07-03T14:22:54","slug":"network-batch-processing","status":"publish","type":"odin-feature","link":"https:\/\/opturo.com\/staging\/odin-features\/network-batch-processing\/","title":{"rendered":"Network Batch Processing"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Current large volume Batch Processing is often limited to one machine, they are run sequentially, have a single point of failure and provide no dashboard to monitor and scale on-demand. Process logging is non-existent or clunky.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Opturo\u2019s Network Processing product provides a robust platform with a decentralized architecture, an Administration Console and one or many Operational Clusters. Following are the components:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>An\u00a0<strong>Operational Cluster<\/strong>\u00a0consists of one \u201cManager\/Broker\u201d and multiple \u201cWorker\u201d daemon processes. Worker daemon within a Cluster can run on one or more machines on the network.<\/li>\n\n\n\n<li>A\u00a0<strong>Manager\/Broker<\/strong>\u00a0within a Cluster maintains a list of \u201cQueues\u201d and distributes \u201cJobs\u201d to Workers.<\/li>\n\n\n\n<li>Each\u00a0<strong>Worker\/Consumer<\/strong>\u00a0daemon subscribes to Jobs on one or more Queues and processes them.<\/li>\n\n\n\n<li>A\u00a0<strong>Batch Process<\/strong>\u00a0with one or more \u201cIterations\u201d or \u201cJobs\u201d, is initiated by a \u201cProducer\u201d which sends a message to the Manager with list of Jobs\/Iterations and a Queue name.<\/li>\n\n\n\n<li>A\u00a0<strong>Job\/Message<\/strong>\u00a0or\u00a0<strong>Iteration<\/strong>\u00a0is one loop of any process, such as calculating analytics on 20 accounts (20 Iterations) or processing 100 files (100 Iterations).<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Using this architecture, the processing of any Batch Process with one or more Jobs\/Iterations can be distributed across multiple machines on a network. This drastically cut run times and improves reliability.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Administration Console allows real-time monitoring current Batch Processes. Provides the ability to scale Workers on-demand to handle larger volumes.<\/p>\n","protected":false},"featured_media":6080,"template":"","odin-feature-category":[],"class_list":["post-1040","odin-feature","type-odin-feature","status-publish","has-post-thumbnail","hentry"],"meta_box":{"wysiwyg_0cxzk7oh9hmf":"<p>Opturo\u2019s Network Processing framework enables high-volume batch processing with a decentralized architecture, real-time monitoring, and on-demand scaling. By distributing workloads across multiple machines, it eliminates single points of failure, reduces processing time, and improves reliability.<\/p>\n","group_k2p5ta854t":[{"opturo-advantage-title":"Integrated with Opturo's ODIN modules","opturo-advantage-body":"<p>New custom Batch Processes are created quickly using ODIN modules which leverages the robust data management\/custom solution\/third-party integration functionality of the ODIN Platform.<\/p>"},{"opturo-advantage-title":"No limitation on scalability","opturo-advantage-body":"<p>There is no known upper limit on scaling a Batch Process using this product. The product can be scaled to meet the demand of a 1K or 1B Iteration Process. The only constraining factor is hardware.<\/p>"},{"opturo-advantage-title":"High Availability and Redundancy","opturo-advantage-body":"<p>No single point of failure. Workers can be designated as back-up Manager\/Broker which is activated if the main Manager\/Broker is offline.<\/p>"},{"opturo-advantage-title":"Web-based Dashboard Manager","opturo-advantage-body":"<p>Real-time monitoring through web-based Administration Console of Batch Processes and machines on the Cluster, allows users to effectively manage the process. System allows scaling on-demand by having the ability to start new Workers on a remote machine.<\/p>"},{"opturo-advantage-title":"Priority Queues","opturo-advantage-body":"<p>Use of Priority Queues allow for time-sensitive Batch Processes to take precedence.<\/p>"},{"opturo-advantage-title":"Detailed Real-time and Historical Logs","opturo-advantage-body":"<p>Provide access to current and Historical Iteration-level logs with real-time machine health statistics.<\/p>"}],"image_advanced_snoqgqw2js":[{"width":150,"height":150,"file":"2025\/06\/Network-Process-1-NB-scaled.png","filesize":181591,"sizes":{"medium":{"file":"Network-Process-1-NB-300x169.png","width":300,"height":169,"mime-type":"image\/png","filesize":9786,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-1-NB-300x169.png"},"large":{"file":"Network-Process-1-NB-1024x576.png","width":1024,"height":576,"mime-type":"image\/png","filesize":64231,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-1-NB-1024x576.png"},"thumbnail":{"file":"Network-Process-1-NB-150x150.png","width":150,"height":150,"mime-type":"image\/png","filesize":4541,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-1-NB-150x150.png"},"medium_large":{"file":"Network-Process-1-NB-768x432.png","width":768,"height":432,"mime-type":"image\/png","filesize":40309,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-1-NB-768x432.png"},"1536x1536":{"file":"Network-Process-1-NB-1536x864.png","width":1536,"height":864,"mime-type":"image\/png","filesize":299425,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-1-NB-1536x864.png"},"2048x2048":{"file":"Network-Process-1-NB-2048x1152.png","width":2048,"height":1152,"mime-type":"image\/png","filesize":452069,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-1-NB-2048x1152.png"}},"image_meta":{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0","keywords":[]},"original_image":"Network-Process-1-NB.png","ID":"6085","name":"Network-Process-1-NB-scaled.png","path":"\/home\/opturoadmin\/opturo_com\/public_html\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-1-NB-scaled.png","url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-1-NB-150x150.png","full_url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-1-NB-scaled.png","title":"Network Process 1 NB","caption":"","description":"","alt":"","srcset":false},{"width":150,"height":150,"file":"2025\/06\/Network-Process-2-NB-scaled.png","filesize":89739,"sizes":{"medium":{"file":"Network-Process-2-NB-300x169.png","width":300,"height":169,"mime-type":"image\/png","filesize":5133,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-2-NB-300x169.png"},"large":{"file":"Network-Process-2-NB-1024x576.png","width":1024,"height":576,"mime-type":"image\/png","filesize":27710,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-2-NB-1024x576.png"},"thumbnail":{"file":"Network-Process-2-NB-150x150.png","width":150,"height":150,"mime-type":"image\/png","filesize":2223,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-2-NB-150x150.png"},"medium_large":{"file":"Network-Process-2-NB-768x432.png","width":768,"height":432,"mime-type":"image\/png","filesize":16723,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-2-NB-768x432.png"},"1536x1536":{"file":"Network-Process-2-NB-1536x864.png","width":1536,"height":864,"mime-type":"image\/png","filesize":126009,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-2-NB-1536x864.png"},"2048x2048":{"file":"Network-Process-2-NB-2048x1152.png","width":2048,"height":1152,"mime-type":"image\/png","filesize":186179,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-2-NB-2048x1152.png"}},"image_meta":{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0","keywords":[]},"original_image":"Network-Process-2-NB.png","ID":"6084","name":"Network-Process-2-NB-scaled.png","path":"\/home\/opturoadmin\/opturo_com\/public_html\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-2-NB-scaled.png","url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-2-NB-150x150.png","full_url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-2-NB-scaled.png","title":"Network Process 2 NB","caption":"","description":"","alt":"","srcset":false},{"width":150,"height":150,"file":"2025\/06\/Network-Process-3-NB-scaled.png","filesize":94517,"sizes":{"medium":{"file":"Network-Process-3-NB-300x169.png","width":300,"height":169,"mime-type":"image\/png","filesize":6040,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-3-NB-300x169.png"},"large":{"file":"Network-Process-3-NB-1024x576.png","width":1024,"height":576,"mime-type":"image\/png","filesize":33173,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-3-NB-1024x576.png"},"thumbnail":{"file":"Network-Process-3-NB-150x150.png","width":150,"height":150,"mime-type":"image\/png","filesize":2553,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-3-NB-150x150.png"},"medium_large":{"file":"Network-Process-3-NB-768x432.png","width":768,"height":432,"mime-type":"image\/png","filesize":20214,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-3-NB-768x432.png"},"1536x1536":{"file":"Network-Process-3-NB-1536x864.png","width":1536,"height":864,"mime-type":"image\/png","filesize":54678,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-3-NB-1536x864.png"},"2048x2048":{"file":"Network-Process-3-NB-2048x1152.png","width":2048,"height":1152,"mime-type":"image\/png","filesize":223160,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-3-NB-2048x1152.png"}},"image_meta":{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0","keywords":[]},"original_image":"Network-Process-3-NB.png","ID":"6083","name":"Network-Process-3-NB-scaled.png","path":"\/home\/opturoadmin\/opturo_com\/public_html\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-3-NB-scaled.png","url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-3-NB-150x150.png","full_url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-3-NB-scaled.png","title":"Network Process 3 NB","caption":"","description":"","alt":"","srcset":false},{"width":150,"height":150,"file":"2025\/06\/Network-Process-6-NB-scaled.png","filesize":89948,"sizes":{"medium":{"file":"Network-Process-6-NB-300x169.png","width":300,"height":169,"mime-type":"image\/png","filesize":6672,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-6-NB-300x169.png"},"large":{"file":"Network-Process-6-NB-1024x576.png","width":1024,"height":576,"mime-type":"image\/png","filesize":75546,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-6-NB-1024x576.png"},"thumbnail":{"file":"Network-Process-6-NB-150x150.png","width":150,"height":150,"mime-type":"image\/png","filesize":3217,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-6-NB-150x150.png"},"medium_large":{"file":"Network-Process-6-NB-768x432.png","width":768,"height":432,"mime-type":"image\/png","filesize":18615,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-6-NB-768x432.png"},"1536x1536":{"file":"Network-Process-6-NB-1536x864.png","width":1536,"height":864,"mime-type":"image\/png","filesize":130635,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-6-NB-1536x864.png"},"2048x2048":{"file":"Network-Process-6-NB-2048x1152.png","width":2048,"height":1152,"mime-type":"image\/png","filesize":195439,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-6-NB-2048x1152.png"}},"image_meta":{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0","keywords":[]},"original_image":"Network-Process-6-NB.png","ID":"6080","name":"Network-Process-6-NB-scaled.png","path":"\/home\/opturoadmin\/opturo_com\/public_html\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-6-NB-scaled.png","url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-6-NB-150x150.png","full_url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-6-NB-scaled.png","title":"Network Process 6 NB","caption":"","description":"","alt":"","srcset":false},{"width":150,"height":150,"file":"2025\/06\/Network-Process-5-NB-scaled.png","filesize":79953,"sizes":{"medium":{"file":"Network-Process-5-NB-300x169.png","width":300,"height":169,"mime-type":"image\/png","filesize":3559,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-5-NB-300x169.png"},"large":{"file":"Network-Process-5-NB-1024x576.png","width":1024,"height":576,"mime-type":"image\/png","filesize":22594,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-5-NB-1024x576.png"},"thumbnail":{"file":"Network-Process-5-NB-150x150.png","width":150,"height":150,"mime-type":"image\/png","filesize":1800,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-5-NB-150x150.png"},"medium_large":{"file":"Network-Process-5-NB-768x432.png","width":768,"height":432,"mime-type":"image\/png","filesize":13886,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-5-NB-768x432.png"},"1536x1536":{"file":"Network-Process-5-NB-1536x864.png","width":1536,"height":864,"mime-type":"image\/png","filesize":41241,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-5-NB-1536x864.png"},"2048x2048":{"file":"Network-Process-5-NB-2048x1152.png","width":2048,"height":1152,"mime-type":"image\/png","filesize":62561,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-5-NB-2048x1152.png"}},"image_meta":{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0","keywords":[]},"original_image":"Network-Process-5-NB.png","ID":"6081","name":"Network-Process-5-NB-scaled.png","path":"\/home\/opturoadmin\/opturo_com\/public_html\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-5-NB-scaled.png","url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-5-NB-150x150.png","full_url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-5-NB-scaled.png","title":"Network Process 5 NB","caption":"","description":"","alt":"","srcset":false},{"width":150,"height":150,"file":"2025\/06\/Network-Process-4-Do-Running-Jobs-NB-scaled.png","filesize":83730,"sizes":{"medium":{"file":"Network-Process-4-Do-Running-Jobs-NB-300x169.png","width":300,"height":169,"mime-type":"image\/png","filesize":6452,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-4-Do-Running-Jobs-NB-300x169.png"},"large":{"file":"Network-Process-4-Do-Running-Jobs-NB-1024x576.png","width":1024,"height":576,"mime-type":"image\/png","filesize":28065,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-4-Do-Running-Jobs-NB-1024x576.png"},"thumbnail":{"file":"Network-Process-4-Do-Running-Jobs-NB-150x150.png","width":150,"height":150,"mime-type":"image\/png","filesize":3122,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-4-Do-Running-Jobs-NB-150x150.png"},"medium_large":{"file":"Network-Process-4-Do-Running-Jobs-NB-768x432.png","width":768,"height":432,"mime-type":"image\/png","filesize":18435,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-4-Do-Running-Jobs-NB-768x432.png"},"1536x1536":{"file":"Network-Process-4-Do-Running-Jobs-NB-1536x864.png","width":1536,"height":864,"mime-type":"image\/png","filesize":126747,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-4-Do-Running-Jobs-NB-1536x864.png"},"2048x2048":{"file":"Network-Process-4-Do-Running-Jobs-NB-2048x1152.png","width":2048,"height":1152,"mime-type":"image\/png","filesize":187690,"url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-4-Do-Running-Jobs-NB-2048x1152.png"}},"image_meta":{"aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0","keywords":[]},"original_image":"Network-Process-4-Do-Running-Jobs-NB.png","ID":"6082","name":"Network-Process-4-Do-Running-Jobs-NB-scaled.png","path":"\/home\/opturoadmin\/opturo_com\/public_html\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-4-Do-Running-Jobs-NB-scaled.png","url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-4-Do-Running-Jobs-NB-150x150.png","full_url":"https:\/\/opturo.com\/staging\/wp-content\/uploads\/2025\/06\/Network-Process-4-Do-Running-Jobs-NB-scaled.png","title":"Network Process 4 Do Running Jobs NB","caption":"","description":"","alt":"","srcset":false}],"order":"8"},"_links":{"self":[{"href":"https:\/\/opturo.com\/staging\/wp-json\/wp\/v2\/odin-feature\/1040","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/opturo.com\/staging\/wp-json\/wp\/v2\/odin-feature"}],"about":[{"href":"https:\/\/opturo.com\/staging\/wp-json\/wp\/v2\/types\/odin-feature"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/opturo.com\/staging\/wp-json\/wp\/v2\/media\/6080"}],"wp:attachment":[{"href":"https:\/\/opturo.com\/staging\/wp-json\/wp\/v2\/media?parent=1040"}],"wp:term":[{"taxonomy":"odin-feature-category","embeddable":true,"href":"https:\/\/opturo.com\/staging\/wp-json\/wp\/v2\/odin-feature-category?post=1040"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}