Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: DataFrame.from_records()'s columns argument doesn't work on Numpy's structured array #59717

Open
3 tasks done
ammar-qazi opened this issue Sep 5, 2024 · 5 comments · May be fixed by #59809
Open
3 tasks done

BUG: DataFrame.from_records()'s columns argument doesn't work on Numpy's structured array #59717

ammar-qazi opened this issue Sep 5, 2024 · 5 comments · May be fixed by #59809
Assignees
@VibavariG
Labels
Bug IO Data IO issues that don't fit into a more specific label

Comments

@ammar-qazi
Copy link
Contributor

ammar-qazi commented Sep 5, 2024 •
edited
Loading

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

data = np.array([
    ('John', 25, 'New York', 50000),
    ('Jane', 30, 'San Francisco', 75000),
    ('Bob', 35, 'Chicago', 65000),
    ('Alice', 28, 'Los Angeles', 60000)
], dtype=[('name', 'U10'), ('age', 'i4'), ('city', 'U15'), ('salary', 'i4')])

df = pd.DataFrame.from_records(data, columns=['salary', 'city', 'age', 'name'])

print(df)

Issue Description

The columns parameter doesn't appear to reorder the columns when the input data is a structured array, even though the documentation says it should. (It does do this for other data types.) In addition, the argument doesn't filter the columns like it does for other input types like dict (a behavior which isn't documented yet, but which will be -- see #59670).

Expected Behavior

According to the discussion in #15319 and #59670, DataFrame.from_records()'s columns argument should allow the users to reorder columns (and include only specific columns) from the Numpy's structured array the way it works for data passed as a dictionary.

Installed Versions

INSTALLED VERSIONS

commit : a671b5a
python : 3.10.12.final.0
python-bits : 64
OS : Linux
OS-release : 6.1.85+
Version : #1 SMP PREEMPT_DYNAMIC Thu Jun 27 21:05:47 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.1.4
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.8.2
setuptools : 71.0.4
pip : 24.1.2
Cython : 3.0.11
pytest : 7.4.4
hypothesis : None
sphinx : 5.0.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.4
html5lib : 1.1
pymysql : None
psycopg2 : 2.9.9
jinja2 : 3.1.4
IPython : 7.34.0
pandas_datareader : 0.10.0
bs4 : 4.12.3
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : 2024.6.1
gcsfs : 2024.6.1
matplotlib : 3.7.1
numba : 0.60.0
numexpr : 2.10.1
odfpy : None
openpyxl : 3.1.5
pandas_gbq : 0.23.1
pyarrow : 14.0.2
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.13.1
sqlalchemy : 2.0.32
tables : 3.8.0
tabulate : 0.9.0
xarray : 2024.6.0
xlrd : 2.0.1
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None
@ammar-qazi ammar-qazi added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 5, 2024
@cjerdonek
Copy link

Since the "filtering" aspect isn't documented yet, it seems like the emphasis in this issue should be on the fact that the columns argument doesn't do any reordering, which is documented. So I would rephrase the issue description to something like this (and provide another example as needed)--

The Columns parameter doesn't appear to reorder the columns when the input data is a structured array, even though the documentation says it should. (It does do this for other data types.) In addition, the argument doesn't filter the columns like it does for other input types like dict (a behavior which isn't documented yet, but which will be -- see #59670).

@cjerdonek cjerdonek mentioned this issue Sep 5, 2024
Open
1 task
@ammar-qazi
Copy link
Contributor Author

@cjerdonek I've adjusted the issue text to reflect this.

@cjerdonek
Copy link

Thanks!

@rhshadrach
Copy link
Member

Thanks for the report, confirmed on main. Further investigations and PRs to fix are welcome!

@rhshadrach rhshadrach added IO Data IO issues that don't fit into a more specific label and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 8, 2024
VibavariG added a commit to VibavariG/pandas that referenced this issue Sep 14, 2024
@VibavariG
BUG: Fix columns param reorder issue - if columns!=None, use passed p…
b5a6648 
…aram (pandas-dev#59717)
@VibavariG VibavariG linked a pull request Sep 15, 2024 that will close this issue
Open
5 tasks
@VibavariG
Copy link

Take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VibavariG VibavariG

Labels
Bug IO Data IO issues that don't fit into a more specific label
Projects
None yet
Milestone
No milestone
4 participants
@cjerdonek @VibavariG @rhshadrach @ammar-qazi