V1: prototype product API 2 CSV with json buffer #38

Thijzer · 2023-05-09T13:47:47Z

This PR is meant to improve the handling API payload data in a better way.
It already introduced the following improvements:

JSON buffer files: quickly save your API date to a buffer, handy when testing or running multiple transformations on the same data, it also separates the slow API fetching so you could test your transformation a buffer data.
Akeneo Header formation : this code makes all the required headers from an attributes list
powerfull converters : more power is giving to the converter, when adding a converter the data is converted in example from API to editable DATA && from editable DATA back to API. Another benifit is you could connect data using 2 distinct converters, for example from API to editable data && from editable DATA to CSV.
the reader and writer can now accept a converter, reader with use the convert option, writer the revert option.
the encoder has been improved
we added a group formatter with separator to explode grouped fields

We are focusing on the following possible workflows ::
-> Fetch some API data, do some actions, respond with API data
-> Fetch some API data, do some actions, respond with API data, over a downloadable CSV
-> Fetch a CSV file, do some actions, respond with API data

All Actions have now Flat version tests so we can't break any of the current logic when we go and implement the new logic.

TODO

we need to adjust our actions to supports a new format of data
we need to discover new types of actions that are more focused on array data.
we need some kind of validation, as a directive in the first place, validation could run a every xTH item.
converter function convert / revert are bad, we have to improve so that we can determine the direction better

tbrankaer · 2023-08-16T12:47:02Z

src/Component/Akeneo/Header/AkeneoHeader.php

            'currencies' => [],
            'has_locale' => false,
            'has_scope' => false,
        ], $context);
    }

-    public function add($code, string $type, array $context = []): void
+    public function addValue($code, string $type, array $context = []): void


Force code for string input.

tbrankaer · 2023-08-16T12:48:58Z

src/Component/Configurator/Configuration.php

    {
+        if ($this->pipeline === null) {


Use is_null function instead?

tbrankaer · 2023-08-16T12:51:06Z

src/Component/Configurator/ConfigurationManager.php

@@ -292,6 +305,11 @@ public function createWriter(array $configuration): ItemWriterInterface
        $factory = $this->factory->getFactory('writer');
        $writer = $factory->createFromConfiguration($configuration, $this->getWorkFileManager());

+        if (isset($configuration['converter'])) {
+            $converter = $this->createConverter($configuration['converter']);
+            $this->config->getPipeline()->line(new RevertPipe($converter));


$converter variable needed? $this->config->getPipeline()->line(new RevertPipe($this->createConverter($configuration['converter'])));

tbrankaer · 2023-08-16T12:51:30Z

src/Component/Configurator/ConfigurationManager.php

@@ -348,6 +366,11 @@ public function createReader(array $configuration)
        $factory = $this->factory->getFactory('reader');
        $reader = $factory->createFromConfiguration($cursor, $configuration, $this->getConfig());

+        if (isset($configuration['converter'])) {
+            $converter = $this->createConverter($configuration['converter']);


Same. Variable needed?

tbrankaer · 2023-08-16T12:59:32Z

src/Component/Converter/AkeneoProductCsvConverter.php

+    ];
+    private $decoder;
+
+    public function convert(array $item): array


Code duplication AkeneoProductApiConverter? Can we use one function for this?

tbrankaer · 2023-08-16T13:01:39Z

src/Component/Decoder/ItemDecoder.php

            case $class instanceof StringFormat:
                $item[$property] = $class->reverseFormat($item[$property]);
                break;
            case $class instanceof ArrayFormat:
-                $item = $class->reverseFormat($item);
+            $item = $class->reverseFormat($item);


tbrankaer · 2023-08-16T13:02:30Z

src/Component/Encoder/ItemEncoder.php

            if (isset($item[$property])) {
                foreach ($matches as $match) {
                    $this->processMatch($item, $property, $match);
                }
+            } else {


Avoid else, use continue instead

tbrankaer · 2023-08-16T13:04:18Z

src/Component/Format/ArrayGroupFormat.php

+            $this->getOption('separator')
+        );
+    }
+


Remove whitespace

tbrankaer · 2023-08-16T13:04:23Z

src/Component/Format/ArrayGroupFormat.php

+    {
+        return $this->implodeKeys($value, $this->getOption('separator'));
+    }
+


Remove whitespace

tbrankaer · 2023-08-16T13:05:10Z

src/Component/Format/ArrayGroupFormat.php

+            $newPath = $path . $key;
+            if (is_array($value) && $value !== []) {
+                $this->implodeKeys($value, $separator, $result, $newPath . $separator);
+            } else {


Use continue instead

tbrankaer · 2023-08-16T13:05:37Z

src/Component/Format/ArrayGroupFormat.php

+    {
+        foreach ($array as $key => $value) {
+            $newPath = $path . $key;
+            if (is_array($value) && $value !== []) {


tbrankaer · 2023-08-16T13:06:55Z

src/Component/Format/ArrayListFormat.php

+            $newPath = $path . $key;
+            if (is_array($value) && $value !== []) {
+                $this->implodeKeys($value, $separator, $result, $newPath . $separator);
+            } else {


Use continue

tbrankaer · 2023-08-16T13:08:17Z

src/Component/Reader/ItemReader.php

                });
+            } elseif (is_string($rowValue) && in_array($rowValue, ['UNIQUE', 'IS_NOT_NUMERIC', 'NOT_EMPTY', 'NOT_NULL'])) {


Use switch with default here?

tbrankaer self-requested a review August 16, 2023 12:42

tbrankaer reviewed Aug 16, 2023

View reviewed changes

src/Component/Configurator/Configuration.php

{

if ($this->pipeline === null) {

Copy link

tbrankaer Aug 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use is_null function instead?

tbrankaer reviewed Aug 16, 2023

View reviewed changes

src/Component/Format/ArrayGroupFormat.php

$this->getOption('separator')

);

}

Copy link

tbrankaer Aug 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove whitespace

tbrankaer reviewed Aug 16, 2023

View reviewed changes

src/Component/Format/ArrayGroupFormat.php

{

return $this->implodeKeys($value, $this->getOption('separator'));

}

Copy link

tbrankaer Aug 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove whitespace

tbrankaer reviewed Aug 16, 2023

View reviewed changes

Thijzer added 6 commits August 18, 2023 11:40

V1: prototype product API 2 CSV with json buffer

aa02556

improv: actions for listed values

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified
Learn about vigilant mode

52691f2

fix: rename action tests

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified
Learn about vigilant mode

6e6ac7e

added: attribute converter

00c09d5

improved the converter process

bf8cde2

added: action tests

3b7e6f2

Thijzer force-pushed the multi-prototype branch from 54f0136 to 3b7e6f2 Compare August 18, 2023 09:42

Thijzer added 2 commits August 18, 2023 11:44

removed: >>head content<<

ba82cf3

added: more tests

9527729

Thijzer changed the base branch from master to obelink-xml-v3 December 13, 2024 14:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

V1: prototype product API 2 CSV with json buffer #38

V1: prototype product API 2 CSV with json buffer #38

Thijzer commented May 9, 2023 •

edited

Loading

tbrankaer Aug 16, 2023

tbrankaer Aug 16, 2023

tbrankaer Aug 16, 2023

tbrankaer Aug 16, 2023

tbrankaer Aug 16, 2023

tbrankaer Aug 16, 2023

tbrankaer Aug 16, 2023

tbrankaer Aug 16, 2023

tbrankaer Aug 16, 2023

tbrankaer Aug 16, 2023

tbrankaer Aug 16, 2023

tbrankaer Aug 16, 2023

tbrankaer Aug 16, 2023

		});
		} elseif (is_string($rowValue) && in_array($rowValue, ['UNIQUE', 'IS_NOT_NUMERIC', 'NOT_EMPTY', 'NOT_NULL'])) {

V1: prototype product API 2 CSV with json buffer #38

Are you sure you want to change the base?

V1: prototype product API 2 CSV with json buffer #38

Conversation

Thijzer commented May 9, 2023 • edited Loading

TODO

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Thijzer commented May 9, 2023 •

edited

Loading