How to write decimals with arbitrary length?

[dburtsev]

Issue Description

I am trying to create a parquet file with NUMERIC(4, 1) column.

Environment Information

• ParquetSharp Version: 16.1.0
• .NET Framework/SDK Version: net8.0
• Operating System: e.g. Windows 10

Steps To Reproduce

`using System.Data;
using ParquetSharp; // dotnet add package ParquetSharp
namespace TestDecimal
{
internal class Program
{
static void Main(string[] args)
{
var decArray = new decimal?[] { };
DataTable dt = new System.Data.DataTable();
DataColumn column = new System.Data.DataColumn();
column.DataType = System.Type.GetType("System.Decimal"); // NUMERIC(4, 1)
column.ColumnName = "Num";
column.DefaultValue = 123.4;
dt.Columns.Add(column);

        int scale = 1;
        int precision = 4;
        var columns = new ParquetSharp.Column[dt.Columns.Count];
        for (int i = 0; i < dt.Columns.Count; i++)
        {
            columns[i] = new ParquetSharp.Column<decimal?>(dt.Columns[i].ColumnName, LogicalType.Decimal(precision, scale)); 
        }
        String filePath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "test.parquet");
        Console.WriteLine("The output file is {0}", filePath);
        using var file = new ParquetFileWriter(filePath, columns);
        using var rowGroup = file.AppendRowGroup();
        decArray = dt.AsEnumerable().Select(d => d.Field<decimal?>(dt.Columns["Num"])).ToArray();
        using (var decWriter = rowGroup.NextColumn().LogicalWriter<decimal?>())
        {
            decWriter.WriteBatch(decArray);
        }
        file.Close();
    }
}

}
`

Expected Behavior

create test.parquet file with NUMERIC(4, 1) column

Additional Context (Optional)

System.NotSupportedException
HResult=0x80131515
Message=only 29 digits of precision is currently supported for decimal type
Source=ParquetSharp
StackTrace:
at ParquetSharp.LogicalWrite2.ValidateDecimalColumn(ColumnDescriptor columnDescriptor) at ParquetSharp.LogicalWrite2.GetConverter(ColumnDescriptor columnDescriptor, ByteBuffer byteBuffer)
at ParquetSharp.LogicalWriteConverterFactory.GetConverter[TLogical,TPhysical](ColumnDescriptor columnDescriptor, ByteBuffer byteBuffer)
at ParquetSharp.LogicalColumnWriter1.Create[TPhysical,TLogical](ColumnWriter columnWriter, Int32 bufferLength) at ParquetSharp.LogicalColumnWriter.Creator.OnColumnDescriptor[TPhysical,TLogical,TElement]() at ParquetSharp.ColumnDescriptor.Apply[TReturn](LogicalTypeFactory typeFactory, Type columnLogicalTypeOverride, Boolean useNesting, IColumnDescriptorVisitor1 visitor)
at ParquetSharp.LogicalColumnWriter.Create(ColumnWriter columnWriter, Int32 bufferLength, Type elementTypeOverride)
at ParquetSharp.LogicalColumnWriter.Create[TElementType](ColumnWriter columnWriter, Int32 bufferLength, Type elementTypeOverride)
at ParquetSharp.ColumnWriter.LogicalWriter[TElement](Int32 bufferLength)
at TestDecimal.Program.Main(String[] args) in C:\Scripts\TestDecimal\Program.cs:line 29

[marcin-krystianc]

Hi @dburtsev, the the functionality to write decimals with arbitrary length is in master (#482) but it is not released yet.