Skip to content

Commit fd91e08

Browse files
committed
Updates
1 parent db7e419 commit fd91e08

7 files changed

+1051
-224
lines changed

.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
./.ipynb_checkpoints

LiteLLM/main.py

+1-5
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,7 @@
33
from litellm import completion_with_retries
44
import os
55

6-
## set ENV variables
7-
os.environ["OPENAI_API_KEY"] = "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"
8-
os.environ["HUGGINGFACE_API_KEY"] = "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"
9-
os.environ["COHERE_API_KEY"] = "stUXMZWW4Og2E6x5XLAOXvLMZjjQwJUcWGHNbJ7c" ## REPLACE THIS
10-
os.environ["GEMINI_API_KEY"] = "AIzaSyDPSL6FngK-O45BgXn18rx5-4eVEJ4HAXw"
6+
## set ENV variables on your machine.
117

128
messages = [{ "content": "Hello, how are you?","role": "user"}]
139

QA_Template/.ipynb_checkpoints/QA - Commonsense-checkpoint.ipynb

+401
Large diffs are not rendered by default.

QA_Template/.ipynb_checkpoints/QA - Spatial-checkpoint.ipynb

+375
Large diffs are not rendered by default.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
{
2+
"Experiment_1":{
3+
"Prompt":'''
4+
You are a road safety scientist at a vehicle company with a background in traffic engineering,
5+
civil engineering and autonomous vehicles. I have a road driving scenario ontology as shown below (in DOT format).
6+
7+
digraph Ontology {
8+
EnvironmentalConditions -> WeatherConditions;
9+
EnvironmentalConditions -> LightingConditions;
10+
EnvironmentalConditions -> RoadSurfaceConditions;
11+
EnvironmentalConditions -> TimeOfDay;
12+
EnvironmentalConditions -> Season;
13+
RoadTopologyAndTrafficInfrastructure -> RoadType;
14+
RoadTopologyAndTrafficInfrastructure -> RoadSigns;
15+
RoadTopologyAndTrafficInfrastructure -> TrafficLights;
16+
RoadTopologyAndTrafficInfrastructure -> PedestrianCrossings;
17+
RoadTopologyAndTrafficInfrastructure -> Junction -> Roundabout;
18+
RoadTopologyAndTrafficInfrastructure -> Junction -> Intersection;
19+
RoadTopologyAndTrafficInfrastructure -> RoadMarkings;
20+
RoadTopologyAndTrafficInfrastructure -> SpeedLimit;
21+
RoadTopologyAndTrafficInfrastructure -> TrafficCongestion;
22+
RoadTopologyAndTrafficInfrastructure -> RoadWorks;
23+
RoadTopologyAndTrafficInfrastructure -> ParkingZones;
24+
RoadTopologyAndTrafficInfrastructure -> Bridge;
25+
RoadTopologyAndTrafficInfrastructure -> Tunnel;
26+
RoadTopologyAndTrafficInfrastructure -> TrafficCameras;
27+
RoadTopologyAndTrafficInfrastructure -> SchoolZones;
28+
TrafficParticipantAndBehavior -> Vehicle -> AutonomousVehicle;
29+
TrafficParticipantAndBehavior -> Vehicle -> ManualVehicle;
30+
TrafficParticipantAndBehavior -> Vehicle -> EmergencyVehicle;
31+
TrafficParticipantAndBehavior -> Vehicle -> CommercialVehicle;
32+
TrafficParticipantAndBehavior -> Vehicle -> Motorcycle;
33+
TrafficParticipantAndBehavior -> Vehicle -> PublicTransportation;
34+
Vehicle -> VehicleSafetyFeatures;
35+
Vehicle -> VehicleSize;
36+
Vehicle -> VehicleType;
37+
Vehicle -> VehicleAge;
38+
Vehicle -> VehicleCondition;
39+
Vehicle -> VehicleLoad;
40+
TrafficParticipantAndBehavior -> Pedestrian;
41+
TrafficParticipantAndBehavior -> Cyclist;
42+
TrafficParticipantAndBehavior -> DriverBehavior;
43+
TrafficParticipantAndBehavior -> PedestrianBehavior;
44+
TrafficParticipantAndBehavior -> CyclistBehavior;
45+
TrafficParticipantAndBehavior -> TrafficRulesAdherence;
46+
TrafficParticipantAndBehavior -> Animal;
47+
TrafficParticipantAndBehavior -> DriverDistraction;
48+
TrafficParticipantAndBehavior -> DriverFatigue;
49+
TrafficParticipantAndBehavior -> DriverExperience;
50+
TrafficParticipantAndBehavior -> DriverVision;
51+
TrafficParticipantAndBehavior -> DriverIntoxication;
52+
TrafficParticipantAndBehavior -> DriverAge;
53+
}
54+
55+
Here is some additional information about multiple-choice question formation as well as Bloom's taxonomy:
56+
57+
A multiple-choice question is a collection of three components aimed at testing a student’s understanding of a
58+
certain topic, given a particular context of what the student is expected to know. The topic, as well as the context
59+
of the topic, will be provided to generate effective multiple-choice questions. The three parts of a
60+
multiple-choice question are as follows: a Stem, a Correct Answer, and three Distractors. There must always be only
61+
one correct answer and only three distractors.
62+
63+
The stem is relevant to the question the student will attempt to answer, as well as the relevant context necessary in order
64+
to answer the question. It may be in the form of a question, an incomplete statement, or a scenario. The stem should
65+
focus on assessing the specific knowledge or concept the question aims to evaluate.
66+
67+
The Correct Answer refers to the correct, undisputable answer to the question.
68+
69+
A Distractor is an incorrect answer to the question in the stem and adheres to the following properties. […] Use
70+
“None of the Above” or “All of the Above” style answer choices sparingly. These answer choices have been shown to, in
71+
general, be less effective at measuring or assessing student understanding.
72+
73+
Multiple-choice questions should be clear, concise, and grammatically correct statements. Make sure the questions are
74+
worded in a way that is easy to understand and does not introduce unnecessary complexity or ambiguity. Students should
75+
be able to understand the questions without confusion. The question should not be too long, and allow most students to
76+
finish in less than five minutes. This means adhering to the following properties.
77+
78+
79+
80+
Blooms’ Taxonomy and Action Verbs:
81+
Multiple-choice questions must be well aligned to the learning objectives they are intended to assess students’
82+
knowledge on. This implies that they must assess skills at the right cognitive level corresponding to the Bloom’s
83+
taxonomy categorization of the learning objective. Bloom’s Taxonomy offers a framework for categorizing the depth of
84+
learning, and it provides guidance on selecting appropriate action verbs when writing learning objectives. Here are the
85+
six levels of Bloom’s taxonomy and their definitions:
86+
87+
Remember - This level involves retrieving, recognizing, and recalling relevant knowledge from long-term memory.
88+
89+
Understand - At this level, learners construct meaning from oral, written, and graphic messages through interpreting,
90+
exemplifying, classifying, summarizing, inferring, comparing, and explaining.
91+
92+
Apply - This level requires learners to carry out or use a procedure through executing or implementing it.
93+
94+
Analyze - At this level, learners break material into constituent parts, determine how the parts relate to one another
95+
and to an overall structure or purpose through differentiating, organizing.
96+
97+
Evaluate - This level involves making judgments based on criteria and standards through checking and critiquing.
98+
99+
Create - At this level, learners put elements together to form a coherent or functional whole, or they reorganize
100+
elements into a new pattern or structure through generating.
101+
102+
103+
Given information about the traffic related ontology, as well as question formation guidelines and Bloom's taxonomy,
104+
would you be able to create multiple-choice questions for EACH TYPE OF INFERENCE provided below that fulfill the MCQ
105+
guidelines as well as Bloom's taxonomy skills. The below example shows the type of inference relation as well as an example
106+
of the types of questions asked with that particular relation. For option D, make sure the option is similar to the answer to the question, however incorrect.
107+
108+
Relation -> Formulated question example
109+
AtLocation -> Where would I not want a fox? A. hen house, B. england, C. mountains, D. ...
110+
Causes -> What is the hopeful result of going to see a play? A. being entertained, B. meet, C. sit, D. ...
111+
CapableOf -> Why would a person put flowers in a room with dirty gym socks? A. smell good, B. many colors, C. continue to grow , D. ...
112+
Antonym -> Someone who had a very bad flight might be given a trip in this to make up for it? A. first class, B. reputable, C. propitious , D. ...
113+
HasSubevent -> How does a person begin to attract another person for reproducing? A. kiss, B. genetic mutation, C. have sex , D. ...
114+
HasPrerequisite -> If I am tilting a drink toward my face, what should I do before the liquid spills over? A. open mouth, B. eat first, C. use glass , D. ...
115+
CausesDesire -> What do parents encourage kids to do when they experience boredom? A. read book, B. sleep, C. travel , D. ...
116+
Desires -> What do all humans want to experience in their own home? A. feel comfortable, B. work hard, C. fall in love , D. ...
117+
PartOf -> What would someone wear to protect themselves from a cannon? A. body armor, B. tank, C. hat , D. ...
118+
HasProperty -> What is a reason to pay your television bill? A. legal, B. obsolete, C. entertaining , D. ...
119+
120+
Generate a traffic scenario from the concepts and relations listed above. Then generate three multiple choice questions in accordance with the details above.
121+
Focus on the Bloom's taxonomy term "understand" detailed above. Focus on the commonsense inference AtLocation detailed above.
122+
Generate only the scenario and the multiple choice questions.
123+
''',
124+
"Response":"""
125+
Scenario:
126+
You are driving an autonomous vehicle on a busy city street during the evening rush hour. The weather is clear, but the sun is setting, causing some glare. The road is a four-lane city street with a speed limit of 35 mph. There are traffic lights at each intersection, pedestrian crossings, and road signs indicating school zones and parking zones. The vehicle is equipped with safety features such as automatic braking, lane-keeping assist, and adaptive cruise control.
127+
128+
Question 1:
129+
Where would you expect to find traffic lights in this scenario?
130+
A. On the highway
131+
B. At each intersection
132+
C. In the school zones
133+
D. In the parking zones
134+
135+
Question 2:
136+
Where would you expect to find pedestrian crossings in this scenario?
137+
A. On the highway
138+
B. At each intersection
139+
C. In the school zones
140+
D. In the parking zones
141+
142+
Question 3:
143+
Where would you expect to find road signs indicating school zones in this scenario?
144+
A. On the highway
145+
B. At each intersection
146+
C. Near schools
147+
D. In the parking zones
148+
"""},
149+
"Experiment_2": {
150+
"Prompt":
151+
}
152+
}

QA_Template/QA - Commonsense.ipynb

+14-3
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,21 @@
22
"cells": [
33
{
44
"cell_type": "code",
5-
"execution_count": 1,
5+
"execution_count": null,
66
"id": "e52cea52",
77
"metadata": {},
8-
"outputs": [],
8+
"outputs": [
9+
{
10+
"ename": "",
11+
"evalue": "",
12+
"output_type": "error",
13+
"traceback": [
14+
"\u001b[1;31mRunning cells with 'c:\\Users\\ishaa\\anaconda3\\envs\\sqa\\python.exe' requires the ipykernel package.\n",
15+
"\u001b[1;31mRun the following command to install 'ipykernel' into the Python environment. \n",
16+
"\u001b[1;31mCommand: 'conda install -p \\ ipykernel --update-deps --force-reinstall'"
17+
]
18+
}
19+
],
920
"source": [
1021
"import guidance\n",
1122
"import sys\n",
@@ -393,7 +404,7 @@
393404
"name": "python",
394405
"nbconvert_exporter": "python",
395406
"pygments_lexer": "ipython3",
396-
"version": "3.11.5"
407+
"version": "3.10.14"
397408
}
398409
},
399410
"nbformat": 4,

0 commit comments

Comments
 (0)