Re-implement SYCL backend parallel_for to improve bandwidth utilization

[mmichel11]

High Level Description
This PR improves hardware bandwidth utilization of oneDPL's SYCL backend parallel for pattern through two ideas:

• Process multiple input iterations per work-item which involves a switch to a nd_range kernel combined with a sub / work group strided indexing approach.
• To generate wide loads for small types, implement a path that vectorizes loads / stores by processing adjacent indices within a single work item. This is combined with the above approach to maximize hardware bandwidth utilization. Vectorization is only applied to fundamental types of size less than 4 (e.g. uint16_t, uint8_t) under a contiguous container.

Implementation Details

• Parallel for bricks have been reworked in the following manner:
  • Each brick contains a pack of ranges within its template parameters to define tuning parameters.
  • The following static integral members are defined (implemented with inheritance):
    • __can_vectorize
    • __preferred_vector_size (1 if __can_vectorize is false)
    • __preferred_iters_per_item
  • The following public member functions are defined
    • __scalar_path (for small input sizes this member function is explicitly called)
    • __vector_path (optional for algorithms that are not vectorizable e.g. binary_search)
    • An overloaded function call operator which dispatches to the appropriate strategy

To implement this approach, the parallel for kernel rewrite from #1870 was adopted with additional changes to handle vectorization paths. Additionally, generic vectorization and strided loop utilities have been defined with the intention for these to be applicable in other portions of the codebase as well. Tests have been expanded to ensure coverage of vectorization paths.

This PR will supersedes #1870. Initially, the plan was to merge this PR into 1870 but after comparing the diff, I believe the most straightforward approach will be to target this directly to main.

[SergeyKopienko]

Why you pass __storage parameter by value?

[mmichel11]

Great catch. I have made this a l-value reference.

[SergeyKopienko]

What is the case when __n < 0 is true?

[mmichel11]

Never if a valid sequence is passed :) I switched to __n == 0.

[SergeyKopienko]

Why class walk_scalar_base declared as class but

template <typename _ExecutionPolicy, typename _F, typename _Range>
struct walk1_vector_or_scalar : public walk_vector_or_scalar_base<_Range>

declared as struct ?

[mmichel11]

I have made them all structs for consistency.

[SergeyKopienko]

1. I think that __raw_ptr isn't very good name because begin() usually linked in mind with iterator. But raw usually is some pointer.
2. Do we really need to have here local variable __raw_ptr ? Can we pass __rng.begin() instead of that variable into __vector_walk call?

[mmichel11]

In the contexts in which we vectorize, begin() does return pointers, but I agree the name is confusing.

I have addressed this in a different way due to a performance issue. With uint8_t types, I found the compiler was not properly vectorizing even when calling begin() on the set of ranges within the kernel leading to performance regressions (about 30% slower than where we should be). Calling begin from the host and passing it to the submitter to use in the kernel resolves the issue and gives us good performance.

Since begin() is called on all ranges and passed through the bricks from the submitter, I have switched from the _Rng naming to _Acc here as the underlying type may not be a range. Additional template types are also needed.

Update
Please see the comment: #1976 (comment). All of the begin() calls in this context have been removed.

[SergeyKopienko]

So now we have 3 entity with defined constexpr static bool __can_vectorize :

1. class walk_vector_or_scalar_base
2. class walk_scalar_base
3. struct __brick_shift_left

Does these constexpr-variables really has different semantic?

And if the semantic of these entities are the same, may be make sense to make some re-design to have only one entity __can_vectorize ?

[SergeyKopienko]

In some moments implementation details remind me tag-dispatching which were designed by @rarutyun.
But with some differences: for example the walk2_vectors_or_scalars has not only information about vectorization or parallelization should be executed, but also two variant of functional staff and operator() with compile-time condition check to run one code or another code.

But what if we instead of two different functions

    template <typename _IsFull, typename _ItemId>
    void
    __vector_path(_IsFull __is_full, const _ItemId __idx, _Range __rng) const
    {
        // This is needed to enable vectorization
        auto __raw_ptr = __rng.begin();
        oneapi::dpl::__par_backend_hetero::__vector_walk<__base_t::__preferred_vector_size>{__n}(__is_full, __idx, __f,
                                                                                                 __raw_ptr);
    }

    // _IsFull is ignored here. We assume that boundary checking has been already performed for this index.
    template <typename _IsFull, typename _ItemId>
    void
    __scalar_path(_IsFull, const _ItemId __idx, _Range __rng) const
    {

        __f(__rng[__idx]);
    }

we will have some two functions with the same name and the format excepting the first parameter type which will be used as some tag ?

Please take a look at __parallel_policy_tag_selector_t for details.

[SergeyKopienko]

One more point: __vector_path and __scalar_path tell me about some path but not imp.
May be better to rename them to ..._impl ?

[danhoeflinger]

First round of review. I've not gotten to all the details yet, but this is enough to be interesting.

[mmichel11]

So now we have 3 entity with defined constexpr static bool __can_vectorize :

1. class walk_vector_or_scalar_base
2. class walk_scalar_base
3. struct __brick_shift_left

Does these constexpr-variables really has different semantic?

And if the semantic of these entities are the same, may be make sense to make some re-design to have only one entity __can_vectorize ?

These three cases are all unique when you consider that they define __can_vectorize, __preferred_vector_size, and __preferred_iters_per_item. These three fields are all tightly coupled, so in my opinion it makes sense to define them together for readability. If we were to define a single __can_vectorize, then I think it would need to function more like a trait class dependent on the provided brick as the brick itself plays a roll in whether not vectorization is possible. This design would not get us much in my opinion as we would still need specializations for the different cases.

The three unique cases I mention are the following:

1. struct walk_vector_or_scalar_base - Vectorization is possible so long as the ranges meet the requirements to be vectorizable. This is then used to determine iters per item and the vector size.
2. struct walk_scalar_base - Vectorization is not possible due to some limitation of the brick. Binary search is a good example since its accesses are non-sequential. The iterations per work item is still set based on the size of the provided ranges.
3. struct __brick_shift_left - This brick has a limitation that prevents vectorization and that only one iteration per item can be processed and is a special case.

[mmichel11]

A point I would like to make and an open question from me after our offline discussion regarding type support.

Vectorization paths are only enabled for arithmetic types as compilers only vectorize through these limited set of types (seems to align with sycl::vec supported types). Even something such as a wrapper struct that wraps around a uint8_t does not generate vector instructions which is I have enforced fundamental types in walk_vector_or_scalar_base. Vectorization is applied to 1 and 2 byte types, and I hypothesize user provided structs will likely be bigger than this.

I do think vectorization through an arbitrary struct is possible so long as it is trivially copyable, and it may be worth investigating performance vectorizing through larger structs with multiple fields. This could be done through vectorizing load / stores of the struct via serializing it into a bytestream and making use of some reinterpret casting to apply any functors to the struct. Careful attention would have to be made to alignment.

Such approach would likely have portability restrictions, be optimized for specific hardware vector instructions (e.g. PVC), and would certainly be a precedent in oneDPL, so I do not think it would be appropriate for our generic implementation but rather in a kernel template for a for_each or transform operation. I plan to mention this in a discussion I plan to create after the milestone.

What are others' thoughts here? With the supported types, I believe the usage of __lazy_ctor_storage can be removed from this PR as default constructability is not an issue (I have prepared a patch to do so).

[danhoeflinger]

Cant this (wont this always) underflow for non-full cases?

[danhoeflinger]

I believe I've tracked down to the stride recommender that this is a std::size_t, at least sometimes.
Also, is it ever not a std::size_t? If it is always a std::size_t then lets call it that rather than a template param.

[mmichel11]

For the non-full case, __idx < __size should always hold which can be seen in the non-full case of __strided_loop. It ensures that we do not dispatch an index that is greater than or equal to the size. Actually the logic I have above this that sets __remaining_elements is unnecessary (the ternary is always false) and can be removed.

Throughout this PR I think I can make all of these brick _Idx types just std::size_t as that is what gets passed through. The original implementations were templates but I think it's unnecessary.

Update
I have made these changes.

[danhoeflinger]

I was unclear about why this is necessary with the logic already inside the __vector_... helpers, but I think it is because we can't reverse data which wasn't loaded / initialized. Instead we flip only inside the first part of the local vector. However, it doesn't seem like this final assignment is changing the offset for the final write like I would expect it to.

I must be missing something else because you in theory could still use __vector_store just with an offset of 0 instead of __output_start or something.

Also, it seems like we should probably be consistent and use __pstl_assign here rather than directly written assignment unless there is a reason not to.

[mmichel11]

This case is tricky. Suppose we have a vector size of 4 with 3 remaining elements in the buffer. _IsFull will be false. These three elements we will want to store at indices 0, 1, 2 after reversing in registers.

If we did a:

oneapi::dpl::__par_backend_hetero::__vector_store<__base_t::__preferred_vector_size>{__n}(
    __is_full, __output_start,
    oneapi::dpl::__par_backend_hetero::__lazy_store_transform_op<oneapi::dpl::__internal::__pstl_assign>{},
    __rng1_vector, __rng2);

here then the vector operation would try to store 4 elements as the gap between n and __output_start (0) is large.

We could replace __n in the __vector_store construction with __remaining_elements which would fix this similar to the vector walk deleter, but when implementing the for loop felt more clear. What makes more sense to you?

Good point on consistency with __pstl_assign. I will address it.