758 matching collectives

Althing Web

About

Des sites WordPress rapides, sur-mesures et éco-conçus, connectés à CiviCRM ou à tout autre service tiers dont vous avez besoin !

Com 360 (Besançon)

About

Vos partenaires pour une communication à 360°

Hedwige

About

Entrez dans la magie du design

HackYourJob Community

About

L'association des développeuses et des développeurs freelance épanouis

focus.

About

sharper together — Stratégie, Marque, Naming, Communication, Social

Français en [langue]

About

Traductions du français en allemand, italien et/ou anglais.

esprit collectif

About

Nous sommes un collectif d’entrepreneurs et designers convaincus que l'intelligence collective et le design doivent accélérer la transition écologique. www.espritcollectif.fr

Neargem

About

Swiss Army Software Engineering

Formatrices et formateurs HackYourJob

About

Construction et animation de formations pour les professionnels de l'IT, par des faiseurs freelances ou assimilés, sur des hard skills et soft skills. Coaching, audit et conseil.

Corporate Voices

About

Le collectif qui vous aide à construire et alimenter votre stratégie pour servir votre marque corporate.

BeBranded

About

BeBranded vous accompagne dans la création de votre site internet et l'automatisation de vos process

Bumb.

About

Nous donnons à votre start-up un air de licorne

Ansea

About

Collaborative Commerce: combining Automation and Generative AI

Cliché

About

Designing the obvious

AMPERE

About

Le collectif spécialisé pour vos projets énergétiques.

LinkingBrand Digital Factory

About

+20% de ROI Digital. Grâce à 100 experts digitaux & aux méthodes des Digital Factory

La Lucarne

About

L'agence qui vous accompagne dans vos projets web

Updev Solutions

About

Booster votre productivité avec l'IA .

Carbonfact logo

Carbonfact

Apparel & FashionSmall


Improving Data Import Efficiency

Implementing a background job platform and introducing parallelism at the application level to enhance performance and significantly reduce computing time.


Carbonfact is developing carbon management software for the fashion industry. The company imports and processes extensive data to provide comprehensive reporting and support brands in reducing their carbon footprint. However, some data imports took over 10 hours and required more than 250 GB of disk space, significantly slowing down the process. This delay left limited time for retries and prevented timely client feedback.

Process

  1. Setup BullMQ
    Introduced BullMQ to run data import jobs asynchronously with retry capabilities.

  2. Identify Slowness
    Conducted a comprehensive analysis to identify bottlenecks and rewrote slow parts of the data import process.

  3. Implement Streaming Approach
    Redesigned the import script to utilize a streaming approach, significantly reducing disk space and memory usage.

  4. Leverage Parallelism
    Rewrote the script to take advantage of parallel processing and fully utilize all available machine cores.

Impact

  • Job Monitoring
    The engineering team can now monitor their jobs through existing tools like Sentry.

  • Easy Job Creation
    Engineers can easily create new jobs using the provided documentation and SDK.

  • Processing Time Reduction
    Reduced data import time from 10 hours to just 30 minutes.

  • Disk Space Reduction
    Reduced disk space usage from 250 GB to just 1 GB.

  • Memory Usage Optimization
    Capped memory usage at 4 GB.

By implementing these changes, Carbonfact successfully streamlined their data import process, improved efficiency, and enhanced customer satisfaction by delivering quicker and more accurate insights.

Members

Expertises

Google Cloud PlatformNestNode