Ideas for professional analytics software.

[grayson]

Hello, as much of you have degrees and some are even real scientists, I have a question. Do you often work with analytics? And if so, how do you think any analytics software could improve. so far, I have made a very basic plotting software. What do you think it should have more?
 

THIS IS NOT AN ADVERTISEMENT! IT HAS NOT EVEN RELEASED AND MIGHT JUST BE A DRAFT!

[exchemist]

41 minutes ago, grayson said:

Hello, as much of you have degrees and some are even real scientists, I have a question. Do you often work with analytics? And if so, how do you think any analytics software could improve. so far, I have made a very basic plotting software. What do you think it should have more?
 

THIS IS NOT AN ADVERTISEMENT! IT HAS NOT EVEN RELEASED AND MIGHT JUST BE A DRAFT!

I think you need to check your neurological reference frame.

[Sensei]

You should do some basic research before jumping into the topic in earnest to the level where you present the script to the public.

For example, have you ever heard of Excel? Open Office Spreadsheet? You load a CSV file into them, for example, one click and you have a ready-made chart with many options to choose from.

 

ps. It's good that you write some stuff. New code every day. Practice makes perfect.

1 hour ago, grayson said:

What do you think it should have more?

Do you have support for different curves?

So far you have shown a straight line.

Why didn't you click the Add Point button and present us with a nice, smooth curve?

Let me guess - this function is not implemented yet..

 

Learn how to draw a Catmull-Rom curve:

https://en.wikipedia.org/wiki/Centripetal_Catmull–Rom_spline

https://en.wikipedia.org/wiki/Cubic_Hermite_spline#Catmull–Rom_spline

(The Catmull-Rom spline passes through all control-points, unlike the Bezier and B-Spline).

 

 

Once the control-points are created, they should be saved and restored each time the user runs the program.

The program should allow loading and saving projects. Load from the resent list.

Undo and redo operations.

 

 

I always start writing an application in the .NET Framework (C++/C#) by creating a GUI and begin by creating the top menu.

Typical setup is:

File: Open, Resent list sub-menu, Save, Save As, (more stuff), Exit

Edit: Undo, Redo, Cut, Copy, Paste, Delete (more stuff)

View: (app context dependent)

Tools: Settings, (more stuff)

Help: (more stuff), About

 

This is basically a standard in the industry that a good UI developer should follow.

 

You do in the Visual Studio/Express. Download for free:

https://www.google.com/search?q=Visual+Studio

After creating a basic menu strip, double-click each entry to create actions in C++/C# code.

Exit and About are the easiest, so I start from them.

Tools > Settings should open another window. Settings depend on the application, and are different in each.

Settings should have tree list control on the left, and on the right context dependent panel which the user switch by clicking on the tree entry.

Edited November 3, 2023 by Sensei

[grayson]

20 minutes ago, Sensei said:

You should do some basic research before jumping into the topic in earnest to the level where you present the script to the public.

For example, have you ever heard of Excel? Open Office Spreadsheet? You load a CSV file into them, for example, one click and you have a ready-made chart with many options to choose from.

 

ps. It's good that you write some stuff. New code every day. Practice makes perfect.

Do you have support for different curves?

So far you have shown a straight line.

Why didn't you click the Add Point button and present us with a nice, smooth curve?

Let me guess - this function is not implemented yet..

 

Learn how to draw a Catmull-Rom curve:

https://en.wikipedia.org/wiki/Centripetal_Catmull–Rom_spline

https://en.wikipedia.org/wiki/Cubic_Hermite_spline#Catmull–Rom_spline

(The Catmull-Rom spline passes through all control-points, unlike the Bezier and B-Spline).

 

 

Once the control-points are created, they should be saved and restored each time the user runs the program.

The program should allow loading and saving projects. Load from the resent list.

Undo and redo operations.

 

 

I always start writing an application in the .NET Framework (C++/C#) by creating a GUI and begin by creating the top menu.

Typical setup is:

File: Open, Resent list sub-menu, Save, Save As, (more stuff), Exit

Edit: Undo, Redo, Cut, Copy, Paste, Delete (more stuff)

View: (app context dependent)

Tools: Settings, (more stuff)

Help: (more stuff), About

 

This is basically a standard in the industry that a good UI developer should follow.

 

Thank you very much!

[Sensei]

The app for making graphs should let the user to enter data, not just click and move with the mouse, the values must be precise, if it is to be 100.0, it cannot be done with the mouse. The mouse dragging procedure would have to have a snap-to-grid functionality - which the user should be able to turn on and off at will. (and edit grid size numeric text control - may be in the toolbox)

This allows the user to click each point on the curve and see the value in the text (numeric) control and can change it.

You can also select multiple points (Shift key) or use the region/lasso selection tool and drag them all at once. Or delete (you need del key event handler).

Hovering the mouse over a point should display the value of the point in a floating bubble. You need some kind of threshold, a few pixels (editable in Tools > Settings), not an exact value (already at Full HD it's hard to hit something, at higher resolutions it would be a nightmare).

Import and export various file formats. So the best way would be to make proprietary SDK to let other developers writing extensions. But it is way way more than your level of expertise (you would need to learn how to make C++/C# DLLs/modules)

 

After creating menustrip, you need to create a toolbox with tools - images in buttons.

So you need "Add", "Delete", "Edit" modes, "Select all", "Deselect all", "Invert selection" tools (all image-buttons).

Have a Mode variable in the code. When you click Add Image button, you set Mode=Add (create enums).

When you click Delete image button, you set Mode=Delete, etc.

Select All, Deselect All, Invert are immediate operations.

So you need in your project a list of control-points. Make C++/C# class for them, with properties: position, value, selection-state (which is altered by Select All etc. tools). Edit tool changes value property. Some other tool for sliding them to change position property.

 

1 hour ago, exchemist said:

I think you need to check your neurological reference frame.

..this is the Computer Science subforum..

 

Edited November 3, 2023 by Sensei

[grayson]

Okay, update. You can now save and load csv files in the program. I cannot show the video, as it is too big of a file, but yah. Pretty cool
 

27 minutes ago, Sensei said:

The app for making graphs should let the user to enter data, not just click and move with the mouse, the values must be precise, if it is to be 100.0, it cannot be done with the mouse. The mouse dragging procedure would have to have a snap-to-grid functionality - which the user should be able to turn on and off at will.

This allows the user to click each point on the curve and see the value in the text (numeric) control and can change it.

You can also select multiple points (Shift key) or use the region/lasso selection tool and drag them all at once. Or delete (you need del key event handler).

Hovering the mouse over a point should display the value of the point in a floating bubble. You need some kind of threshold, a few pixels (editable in Tools > Settings), not an exact value (already at Full HD it's hard to hit something, at higher resolutions it would be a nightmare).

Import and export various file formats. So the best way would be to make proprietary SDK to let other developers writing extensions. But it is way way more than your level of expertise (you would need to learn how to make C++/C# DLLs/modules)

 

And yes, it does let you enter data. The way it works, is that you add numbers into the input boxes and then when you press add point it converts the StringVar (A tkinter specific variable type) into a float and than you have to work from there by pressing plot.

[Sensei]

24 minutes ago, grayson said:

Okay, update. You can now save and load csv files in the program. I cannot show the video, as it is too big of a file, but yah. Pretty cool

BTW, CSVs are not as easy to use as you think. Files can have characters other than a comma as a separator (that's why the Open Office Spreadsheet asks the user what character is a "comma" - so you need entire window ("panel"/"GUI") to handle it) and there is a problem with escaping that character (double quotes in some fields). People doing a simple cols = string.split( "," ) create the buggy code.

 

Example CSV:

"1.0","2",3,"4,0"

"1.1","2,1",4,"5,1"

Commas are part of the record field.

Using cols = string.split( "," ) will cause an error in the code.

 

A simple solution is to use a CSV-compatible library. But nevertheless in-depth you need to test it on unusual files (like above CSV)..

 

Edited November 3, 2023 by Sensei

[grayson]

3 minutes ago, Sensei said:

BTW, CSVs are not as easy to use as you think. Files can have characters other than a comma as a separator (that's why the Open Office Spreadsheet asks the user what character is a "comma" - so you need entire window to handle it) and there is a problem with escaping that character (double quotes in some fields). People doing a simple cols = string.split( "," ) create the buggy code.

 

Example CSV:

"1.0","2",3,"4,0"

"1.1","2,1",4,"5,1"

Commas are part of the record field.

Using cols = string.split( "," ) will cause an error in the code.

 

X,Y
5.0,1.0
5.0,2.0
5.0,3.0
That is how it stores the x and y values. That is an example from one of the csv files I saved.

[Sensei]

Just now, grayson said:

X,Y
5.0,1.0
5.0,2.0
5.0,3.0
That is how it stores the x and y values. That is an example from one of the csv files I saved.

If there is a double-quote character in the field, commas (or "commas") are ignored until the next ending double-quote character.

Add quotes, add commas in them, and try loading again..

 

Some countries use a period as a decimal separator and some use a comma as a decimal separator.

https://en.wikipedia.org/wiki/Decimal_separator

 

 

 

By default, computer languages use the user's local settings and the user's country settings when writing and parsing floating-point numbers.

If a CSV, XML, HTML, etc. file is created, it will have an incompatible floating point/number format, and trying to load it in that form in another country will cause problems.

 

For CSV, this will cause problems with simple/buggy CSV parsers, such as line.split( "," )

 

 

When each field in the CSV is in double-quotes, the parser can automatically detect where the separators are and whether we are importing data from a country with a different format than in the USA-UK-Australia-China and a few Africans countries.

 

 

ps. Add padding/margins between controls. Reduce the inner padding from text to border. Text buttons should not glue together. (Image buttons can, if there is image inside of them without borders)

 

[grayson]

46 minutes ago, Sensei said:

If there is a double-quote character in the field, commas (or "commas") are ignored until the next ending double-quote character.

Add quotes, add commas in them, and try loading again..

 

Some countries use a period as a decimal separator and some use a comma as a decimal separator.

https://en.wikipedia.org/wiki/Decimal_separator

 

 

 

By default, computer languages use the user's local settings and the user's country settings when writing and parsing floating-point numbers.

If a CSV, XML, HTML, etc. file is created, it will have an incompatible floating point/number format, and trying to load it in that form in another country will cause problems.

 

For CSV, this will cause problems with simple/buggy CSV parsers, such as line.split( "," )

 

Wait, which countries use the period as a seperator? I do not know how to change that. I just used the csv module.

[Sensei]

3 minutes ago, grayson said:

Wait, which countries use the period as a seperator? I do not know how to change that. I just used the csv module.

I provided a link in previous messages. A map can be found there..

 

Decimal separator can be dot or comma or else.

 

CSV usually use comma as separator. But it is not hardcoded. It can be other character.

Comma decimal separator conflicts with comma separator in CSV (if parser is buggy).

 

 

Download OpenOffice Spreadsheet, make some CSV text file, and try to load, try to save in CSV, it will ask what character do you want as column separator. It can be anything.

 

[grayson]

I can now show you guys all the new features, just in image form though.

[Sensei]

..but where these data is loaded from the CSV...? There should be a curve, not a straight line....

 

Edited November 4, 2023 by Sensei

[grayson]

4 minutes ago, Sensei said:

..but where these data is loaded from the CSV...? There should be a curve, not a straight line....

 

This data is not loaded from a csv. It was manually put in. And I am going to start working on the curve soon.

[Sensei]

7 hours ago, grayson said:

This data is not loaded from a csv. It was manually put in. And I am going to start working on the curve soon.

An intermediate solution should be to simply draw straight lines between the control points.

 

[grayson]

Update: I have not figured out the curves yet, but I have added dots in-between each line you plot so that you can keep track of where you plotted the lines. Before that, I tried adding 3d functionality but I couldn't figure out how to put it into my code.

Another update: I made an algorithm for predicting the next value. It works pretty well, but not PRETTY well. I will show you it right now:

 Ignore the music. For some reason, it records the audio coming from my computer.

 

[iNow]

What is the gap in the current market that you believe this helps fill?

[grayson]

21 minutes ago, iNow said:

What is the gap in the current market that you believe this helps fill?

Good question. The answer is I do not know yet. I am actively trying to figure this out tho

UPDATE: I have made an algorithm! Here is how it works: First, you set an energy threshold. than it basically iterates over the x and y list and for each time that a thing appears, it appends that to x and y. (Which now that I think of it, it is just the x and y list) Than, it selects two things, best_x/y and cx/y.  It calculates the accuracy by taking the absolute value of cyx - best_x/y. If the acuracy is under the threshold, it adds the cx/y to the last value of x/y. This is an algorithm, and while not the best, it does its job. The only problem is that it tends to only go to the top right and lower left. Idk why, but it works.

[iNow]

Look up companies like Tableau, MS Power BI, Fusion Charts, Domo, Google Cloud Analytics, SAP Analytics Cloud, and the 15 new ones that pop up every 3 days. 

[Ghideon]

On 11/3/2023 at 9:07 PM, grayson said:

Do you often work with analytics?

Yes, on a daily basis.

On 11/3/2023 at 9:07 PM, grayson said:

And if so, how do you think any analytics software could improve.

Some ideas: I find myself spending considerate amount of time on data preparation to ensure the data is accurate, consistent, and ready for analysis. This is an area that is complicated and I see room for improvement. Also, in relation to data governance, Data provenance (documentation of the origins, custody, and transformations that a dataset has undergone) is another area that may be improved. A naive & practical example; assume a manager makes a decision based on numbers and diagrams in a spread sheet. Performing reviews of the decision processes and validating the data used in the decision could benefit from better functionality in the software used in the analytics. 

I am sure all the building blocks for the improvements already exists and that integrating them is a challenge. 

[grayson]

6 hours ago, Ghideon said:

Yes, on a daily basis.

Some ideas: I find myself spending considerate amount of time on data preparation to ensure the data is accurate, consistent, and ready for analysis. This is an area that is complicated and I see room for improvement. Also, in relation to data governance, Data provenance (documentation of the origins, custody, and transformations that a dataset has undergone) is another area that may be improved. A naive & practical example; assume a manager makes a decision based on numbers and diagrams in a spread sheet. Performing reviews of the decision processes and validating the data used in the decision could benefit from better functionality in the software used in the analytics. 

I am sure all the building blocks for the improvements already exists and that integrating them is a challenge. 

When you say data preparation do you mean like natural language processing? I can't do that, but I can load in csv files, which can be made from spreadsheets. Plus, you dont have to name the x and y x and y, you can name them anything you want. So You can work with spreadsheets and visualise them in here, but not natural language processing.

[Ghideon]

2 hours ago, grayson said:

When you say data preparation do you mean like natural language processing?

No. Here are a few steps that may be associated with data preparation, none of these requires* NLP.

(Locating Data: Identifying data sources.)
(Collecting Data: Gathering or importing data.)
(Data Ingestion: Obtaining and importing data for use or storage.)
Data Cleaning: Correcting or removing inaccuracies and inconsistencies.
Data Validation: Ensuring accuracy, consistency, and relevancy.
Data Transformation: Converting data to a suitable format or structure.
Data Integration: Combining data from various sources into a unified view.
(Data Storage: Storing clean, ready data for future use.)

Those within parentesis I consider of secondary importance to the software you describe and implemented by other tools. But I consider them important for the analysis I think you have in mind and may give a more complete view of steps involved.

 

*) NLP could of course be incorporated; I do expect the current trend regarding AI (including NLP) and LLM's to affect vendors of analytics software and that is best discussed in a separate thread.